Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Update notebook to work on Colab

Omar Khattabf7d2904b9fc9
diff --git a/intro.ipynb b/intro.ipynb
index 294f2acee..c60a69708 100644
--- a/intro.ipynb
+++ b/intro.ipynb
@@ -5,7 +5,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## 🤝🎓 𝗗𝗦𝗣: 𝗗EMONSTRATE–𝗦EARCH–𝗣REDICT\n",
+ "## 🎓 𝗗𝗦𝗣: 𝗗EMONSTRATE–𝗦EARCH–𝗣REDICT\n",
"\n",
"This notebook illustrates the Demonstrate–Search–Predict (𝗗𝗦𝗣) framework on the multi-hop question answering (QA) task.\n",
"\n",
@@ -17,15 +17,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Setting Up\n",
- "\n",
- "We'll start by setting up the language model (LM) and retrieval model (RM).\n",
- "\n",
- "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",
- "\n",
- "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",
+ "### Installation\n",
"\n",
- "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"
+ "If you haven't installed **DSP** already, let's do that."
]
},
{
@@ -37,12 +31,171 @@
"name": "stdout",
"output_type": "stream",
"text": [
+ "Requirement already satisfied: dsp-ml in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (0.1.1)\n",
+ "Requirement already satisfied: spacy in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (3.5.0)\n",
+ "Requirement already satisfied: backoff in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (2.2.1)\n",
+ "Requirement already satisfied: jupyter in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (1.0.0)\n",
+ "Requirement already satisfied: pandas in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (1.5.3)\n",
+ "Requirement already satisfied: joblib in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (1.2.0)\n",
+ "Requirement already satisfied: openai in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (0.26.1)\n",
+ "Requirement already satisfied: regex in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (2022.10.31)\n",
+ "Requirement already satisfied: ipywidgets in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter->dsp-ml) (8.0.4)\n",
+ "Requirement already satisfied: notebook in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter->dsp-ml) (6.5.2)\n",
+ "Requirement already satisfied: ipykernel in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter->dsp-ml) (6.20.2)\n",
+ "Requirement already satisfied: jupyter-console in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter->dsp-ml) (6.4.4)\n",
+ "Requirement already satisfied: qtconsole in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter->dsp-ml) (5.4.0)\n",
+ "Requirement already satisfied: nbconvert in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter->dsp-ml) (7.2.8)\n",
+ "Requirement already satisfied: requests>=2.20 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from openai->dsp-ml) (2.28.2)\n",
+ "Requirement already satisfied: tqdm in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from openai->dsp-ml) (4.64.1)\n",
+ "Requirement already satisfied: aiohttp in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from openai->dsp-ml) (3.8.3)\n",
+ "Requirement already satisfied: pytz>=2020.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from pandas->dsp-ml) (2022.7.1)\n",
+ "Requirement already satisfied: numpy>=1.20.3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from pandas->dsp-ml) (1.24.1)\n",
+ "Requirement already satisfied: python-dateutil>=2.8.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from pandas->dsp-ml) (2.8.2)\n",
+ "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (2.4.5)\n",
+ "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (3.0.11)\n",
+ "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (1.10.4)\n",
+ "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (2.0.8)\n",
+ "Requirement already satisfied: thinc<8.2.0,>=8.1.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (8.1.7)\n",
+ "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (1.0.9)\n",
+ "Requirement already satisfied: setuptools in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (65.6.3)\n",
+ "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (6.3.0)\n",
+ "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (2.0.7)\n",
+ "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (1.0.4)\n",
+ "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (3.3.0)\n",
+ "Requirement already satisfied: packaging>=20.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (23.0)\n",
+ "Requirement already satisfied: typer<0.8.0,>=0.3.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (0.7.0)\n",
+ "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (3.0.8)\n",
+ "Requirement already satisfied: jinja2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (3.1.2)\n",
+ "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (1.1.1)\n",
+ "Requirement already satisfied: pathy>=0.10.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (0.10.1)\n",
+ "Requirement already satisfied: typing-extensions>=4.2.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4->spacy->dsp-ml) (4.4.0)\n",
+ "Requirement already satisfied: six>=1.5 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from python-dateutil>=2.8.1->pandas->dsp-ml) (1.16.0)\n",
+ "Requirement already satisfied: idna<4,>=2.5 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from requests>=2.20->openai->dsp-ml) (3.4)\n",
+ "Requirement already satisfied: charset-normalizer<4,>=2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from requests>=2.20->openai->dsp-ml) (2.1.1)\n",
+ "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from requests>=2.20->openai->dsp-ml) (1.26.14)\n",
+ "Requirement already satisfied: certifi>=2017.4.17 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from requests>=2.20->openai->dsp-ml) (2022.12.7)\n",
+ "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from thinc<8.2.0,>=8.1.0->spacy->dsp-ml) (0.7.9)\n",
+ "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from thinc<8.2.0,>=8.1.0->spacy->dsp-ml) (0.0.4)\n",
+ "Requirement already satisfied: click<9.0.0,>=7.1.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from typer<0.8.0,>=0.3.0->spacy->dsp-ml) (8.1.3)\n",
+ "Requirement already satisfied: frozenlist>=1.1.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from aiohttp->openai->dsp-ml) (1.3.3)\n",
+ "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from aiohttp->openai->dsp-ml) (4.0.2)\n",
+ "Requirement already satisfied: aiosignal>=1.1.2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from aiohttp->openai->dsp-ml) (1.3.1)\n",
+ "Requirement already satisfied: attrs>=17.3.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from aiohttp->openai->dsp-ml) (22.2.0)\n",
+ "Requirement already satisfied: yarl<2.0,>=1.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from aiohttp->openai->dsp-ml) (1.8.2)\n",
+ "Requirement already satisfied: multidict<7.0,>=4.5 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from aiohttp->openai->dsp-ml) (6.0.4)\n",
+ "Requirement already satisfied: jupyter-client>=6.1.12 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (7.4.9)\n",
+ "Requirement already satisfied: pyzmq>=17 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (25.0.0)\n",
+ "Requirement already satisfied: tornado>=6.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (6.2)\n",
+ "Requirement already satisfied: nest-asyncio in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (1.5.6)\n",
+ "Requirement already satisfied: debugpy>=1.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (1.6.5)\n",
+ "Requirement already satisfied: traitlets>=5.4.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (5.8.1)\n",
+ "Requirement already satisfied: ipython>=7.23.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (8.8.0)\n",
+ "Requirement already satisfied: comm>=0.1.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (0.1.2)\n",
+ "Requirement already satisfied: psutil in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (5.9.4)\n",
+ "Requirement already satisfied: matplotlib-inline>=0.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (0.1.6)\n",
+ "Requirement already satisfied: jupyterlab-widgets~=3.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipywidgets->jupyter->dsp-ml) (3.0.5)\n",
+ "Requirement already satisfied: widgetsnbextension~=4.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipywidgets->jupyter->dsp-ml) (4.0.5)\n",
+ "Requirement already satisfied: MarkupSafe>=2.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jinja2->spacy->dsp-ml) (2.1.2)\n",
+ "Requirement already satisfied: pygments in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-console->jupyter->dsp-ml) (2.14.0)\n",
+ "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-console->jupyter->dsp-ml) (3.0.36)\n",
+ "Requirement already satisfied: bleach in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (5.0.1)\n",
+ "Requirement already satisfied: nbclient>=0.5.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (0.7.2)\n",
+ "Requirement already satisfied: mistune<3,>=2.0.3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (2.0.4)\n",
+ "Requirement already satisfied: defusedxml in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (0.7.1)\n",
+ "Requirement already satisfied: nbformat>=5.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (5.7.3)\n",
+ "Requirement already satisfied: importlib-metadata>=3.6 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (6.0.0)\n",
+ "Requirement already satisfied: jupyter-core>=4.7 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (5.1.3)\n",
+ "Requirement already satisfied: pandocfilters>=1.4.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (1.5.0)\n",
+ "Requirement already satisfied: jupyterlab-pygments in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (0.2.2)\n",
+ "Requirement already satisfied: beautifulsoup4 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (4.11.1)\n",
+ "Requirement already satisfied: tinycss2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (1.2.1)\n",
+ "Requirement already satisfied: Send2Trash>=1.8.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from notebook->jupyter->dsp-ml) (1.8.0)\n",
+ "Requirement already satisfied: prometheus-client in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from notebook->jupyter->dsp-ml) (0.15.0)\n",
+ "Requirement already satisfied: terminado>=0.8.3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from notebook->jupyter->dsp-ml) (0.17.1)\n",
+ "Requirement already satisfied: nbclassic>=0.4.7 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from notebook->jupyter->dsp-ml) (0.4.8)\n",
+ "Requirement already satisfied: argon2-cffi in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from notebook->jupyter->dsp-ml) (21.3.0)\n",
+ "Requirement already satisfied: ipython-genutils in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from notebook->jupyter->dsp-ml) (0.2.0)\n",
+ "Requirement already satisfied: qtpy>=2.0.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from qtconsole->jupyter->dsp-ml) (2.3.0)\n",
+ "Requirement already satisfied: zipp>=0.5 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from importlib-metadata>=3.6->nbconvert->jupyter->dsp-ml) (3.11.0)\n",
+ "Requirement already satisfied: backcall in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (0.2.0)\n",
+ "Requirement already satisfied: pexpect>4.3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (4.8.0)\n",
+ "Requirement already satisfied: pickleshare in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (0.7.5)\n",
+ "Requirement already satisfied: stack-data in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (0.6.2)\n",
+ "Requirement already satisfied: decorator in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (5.1.1)\n",
+ "Requirement already satisfied: jedi>=0.16 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (0.18.2)\n",
+ "Requirement already satisfied: entrypoints in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-client>=6.1.12->ipykernel->jupyter->dsp-ml) (0.4)\n",
+ "Requirement already satisfied: platformdirs>=2.5 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-core>=4.7->nbconvert->jupyter->dsp-ml) (2.6.2)\n",
+ "Requirement already satisfied: jupyter-server>=1.8 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (2.1.0)\n",
+ "Requirement already satisfied: notebook-shim>=0.1.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (0.2.2)\n",
+ "Requirement already satisfied: jsonschema>=2.6 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbformat>=5.1->nbconvert->jupyter->dsp-ml) (4.17.3)\n",
+ "Requirement already satisfied: fastjsonschema in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbformat>=5.1->nbconvert->jupyter->dsp-ml) (2.16.2)\n",
+ "Requirement already satisfied: wcwidth in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->jupyter-console->jupyter->dsp-ml) (0.2.6)\n",
+ "Requirement already satisfied: ptyprocess in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from terminado>=0.8.3->notebook->jupyter->dsp-ml) (0.7.0)\n",
+ "Requirement already satisfied: argon2-cffi-bindings in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from argon2-cffi->notebook->jupyter->dsp-ml) (21.2.0)\n",
+ "Requirement already satisfied: soupsieve>1.2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from beautifulsoup4->nbconvert->jupyter->dsp-ml) (2.3.2.post1)\n",
+ "Requirement already satisfied: webencodings in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from bleach->nbconvert->jupyter->dsp-ml) (0.5.1)\n",
+ "Requirement already satisfied: parso<0.9.0,>=0.8.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jedi>=0.16->ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (0.8.3)\n",
+ "Requirement already satisfied: importlib-resources>=1.4.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (5.10.2)\n",
+ "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (0.19.3)\n",
+ "Requirement already satisfied: pkgutil-resolve-name>=1.3.10 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (1.3.10)\n",
+ "Requirement already satisfied: websocket-client in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (1.4.2)\n",
+ "Requirement already satisfied: jupyter-events>=0.4.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (0.6.3)\n",
+ "Requirement already satisfied: anyio<4,>=3.1.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (3.6.2)\n",
+ "Requirement already satisfied: jupyter-server-terminals in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (0.4.4)\n",
+ "Requirement already satisfied: cffi>=1.0.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from argon2-cffi-bindings->argon2-cffi->notebook->jupyter->dsp-ml) (1.15.1)\n",
+ "Requirement already satisfied: asttokens>=2.1.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from stack-data->ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (2.2.1)\n",
+ "Requirement already satisfied: pure-eval in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from stack-data->ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (0.2.2)\n",
+ "Requirement already satisfied: executing>=1.2.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from stack-data->ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (1.2.0)\n",
+ "Requirement already satisfied: sniffio>=1.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from anyio<4,>=3.1.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (1.3.0)\n",
+ "Requirement already satisfied: pycparser in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->notebook->jupyter->dsp-ml) (2.21)\n",
+ "Requirement already satisfied: rfc3339-validator in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-events>=0.4.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (0.1.4)\n",
+ "Requirement already satisfied: python-json-logger>=2.0.4 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-events>=0.4.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (2.0.4)\n",
+ "Requirement already satisfied: pyyaml>=5.3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-events>=0.4.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (6.0)\n",
+ "Requirement already satisfied: rfc3986-validator>=0.1.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-events>=0.4.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (0.1.1)\n",
+ "Requirement already satisfied: fqdn in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (1.5.1)\n",
+ "Requirement already satisfied: uri-template in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (1.2.0)\n",
+ "Requirement already satisfied: webcolors>=1.11 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (1.12)\n",
+ "Requirement already satisfied: isoduration in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (20.11.0)\n",
+ "Requirement already satisfied: jsonpointer>1.13 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (2.3)\n",
+ "Requirement already satisfied: arrow>=0.15.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from isoduration->jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (1.2.3)\n",
"env: DSP_NOTEBOOK_CACHEDIR=cache\n"
]
}
],
"source": [
- "%set_env DSP_NOTEBOOK_CACHEDIR cache\n",
+ "!pip install dsp-ml\n",
+ "\n",
+ "try:\n",
+ " import google.colab # When on google Colab, let's clone the notebook so we download the cache.\n",
+ " !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",
+ " %set_env DSP_NOTEBOOK_CACHEDIR dsp/cache\n",
+ "\n",
+ "except:\n",
+ " %set_env DSP_NOTEBOOK_CACHEDIR cache # Set up the cache for the examples below."
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Setting Up\n",
+ "\n",
+ "We'll start by setting up the language model (LM) and retrieval model (RM).\n",
+ "\n",
+ "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",
+ "\n",
+ "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",
+ "\n",
+ "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
"%load_ext autoreload\n",
"%autoreload 2\n",
"\n",
@@ -74,7 +227,7 @@
},
{
"cell_type": "code",
- "execution_count": 2,
+ "execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
@@ -98,7 +251,7 @@
},
{
"cell_type": "code",
- "execution_count": 3,
+ "execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
@@ -135,7 +288,7 @@
},
{
"cell_type": "code",
- "execution_count": 4,
+ "execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
@@ -155,7 +308,7 @@
},
{
"cell_type": "code",
- "execution_count": 5,
+ "execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
@@ -177,7 +330,7 @@
},
{
"cell_type": "code",
- "execution_count": 6,
+ "execution_count": 7,
"metadata": {},
"outputs": [
{
@@ -204,7 +357,7 @@
},
{
"cell_type": "code",
- "execution_count": 7,
+ "execution_count": 8,
"metadata": {},
"outputs": [
{
@@ -268,14 +421,14 @@
},
{
"cell_type": "code",
- "execution_count": 8,
+ "execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
- "100%|██████████| 13/13 [00:00<00:00, 354.31it/s]\n"
+ "100%|██████████| 13/13 [00:00<00:00, 384.43it/s]"
]
},
{
@@ -285,124 +438,131 @@
"Answered 4 / 13 correctly.\n"
]
},
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\n"
+ ]
+ },
{
"data": {
"text/html": [
"<style type=\"text/css\">\n",
- "#T_070f8_ th {\n",
+ "#T_364bd th {\n",
" text-align: left;\n",
"}\n",
- "#T_070f8_ td {\n",
+ "#T_364bd td {\n",
" text-align: left;\n",
"}\n",
"</style>\n",
- "<table id=\"T_070f8_\">\n",
+ "<table id=\"T_364bd\">\n",
" <thead>\n",
" <tr>\n",
" <th class=\"blank level0\" >&nbsp;</th>\n",
- " <th class=\"col_heading level0 col0\" >question</th>\n",
- " <th class=\"col_heading level0 col1\" >answer</th>\n",
- " <th class=\"col_heading level0 col2\" >prediction</th>\n",
- " <th class=\"col_heading level0 col3\" >correct</th>\n",
+ " <th id=\"T_364bd_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
+ " <th id=\"T_364bd_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
+ " <th id=\"T_364bd_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
+ " <th id=\"T_364bd_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
- " <th id=\"T_070f8_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_070f8_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_070f8_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_070f8_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_070f8_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
+ " <th id=\"T_364bd_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
+ " <td id=\"T_364bd_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
+ " <td id=\"T_364bd_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
+ " <td id=\"T_364bd_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
+ " <td id=\"T_364bd_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_070f8_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_070f8_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_070f8_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_070f8_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
- " <td id=\"T_070f8_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
+ " <th id=\"T_364bd_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
+ " <td id=\"T_364bd_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
+ " <td id=\"T_364bd_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
+ " <td id=\"T_364bd_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
+ " <td id=\"T_364bd_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_070f8_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_070f8_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_070f8_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_070f8_row2_col2\" class=\"data row2 col2\" >Melbourne, Australia</td>\n",
- " <td id=\"T_070f8_row2_col3\" class=\"data row2 col3\" >❌</td>\n",
+ " <th id=\"T_364bd_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
+ " <td id=\"T_364bd_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
+ " <td id=\"T_364bd_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
+ " <td id=\"T_364bd_row2_col2\" class=\"data row2 col2\" >Melbourne, Australia</td>\n",
+ " <td id=\"T_364bd_row2_col3\" class=\"data row2 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_070f8_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_070f8_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_070f8_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_070f8_row3_col2\" class=\"data row3 col2\" >1966</td>\n",
- " <td id=\"T_070f8_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
+ " <th id=\"T_364bd_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
+ " <td id=\"T_364bd_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
+ " <td id=\"T_364bd_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
+ " <td id=\"T_364bd_row3_col2\" class=\"data row3 col2\" >1966</td>\n",
+ " <td id=\"T_364bd_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_070f8_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_070f8_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_070f8_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_070f8_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
- " <td id=\"T_070f8_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
+ " <th id=\"T_364bd_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
+ " <td id=\"T_364bd_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
+ " <td id=\"T_364bd_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
+ " <td id=\"T_364bd_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
+ " <td id=\"T_364bd_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_070f8_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_070f8_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_070f8_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_070f8_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_070f8_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
+ " <th id=\"T_364bd_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
+ " <td id=\"T_364bd_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
+ " <td id=\"T_364bd_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
+ " <td id=\"T_364bd_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
+ " <td id=\"T_364bd_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_070f8_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_070f8_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_070f8_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_070f8_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
- " <td id=\"T_070f8_row6_col3\" class=\"data row6 col3\" >❌</td>\n",
+ " <th id=\"T_364bd_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
+ " <td id=\"T_364bd_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
+ " <td id=\"T_364bd_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
+ " <td id=\"T_364bd_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
+ " <td id=\"T_364bd_row6_col3\" class=\"data row6 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_070f8_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_070f8_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_070f8_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_070f8_row7_col2\" class=\"data row7 col2\" >July 3, 1961</td>\n",
- " <td id=\"T_070f8_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
+ " <th id=\"T_364bd_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
+ " <td id=\"T_364bd_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
+ " <td id=\"T_364bd_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
+ " <td id=\"T_364bd_row7_col2\" class=\"data row7 col2\" >July 3, 1961</td>\n",
+ " <td id=\"T_364bd_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_070f8_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_070f8_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_070f8_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_070f8_row8_col2\" class=\"data row8 col2\" >Susquehanna River</td>\n",
- " <td id=\"T_070f8_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
+ " <th id=\"T_364bd_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
+ " <td id=\"T_364bd_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
+ " <td id=\"T_364bd_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
+ " <td id=\"T_364bd_row8_col2\" class=\"data row8 col2\" >Susquehanna River</td>\n",
+ " <td id=\"T_364bd_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_070f8_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_070f8_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_070f8_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_070f8_row9_col2\" class=\"data row9 col2\" >Mars exploration</td>\n",
- " <td id=\"T_070f8_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
+ " <th id=\"T_364bd_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
+ " <td id=\"T_364bd_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
+ " <td id=\"T_364bd_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
+ " <td id=\"T_364bd_row9_col2\" class=\"data row9 col2\" >Mars exploration</td>\n",
+ " <td id=\"T_364bd_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_070f8_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_070f8_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_070f8_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_070f8_row10_col2\" class=\"data row10 col2\" >Hong Kong</td>\n",
- " <td id=\"T_070f8_row10_col3\" class=\"data row10 col3\" >❌</td>\n",
+ " <th id=\"T_364bd_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
+ " <td id=\"T_364bd_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
+ " <td id=\"T_364bd_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
+ " <td id=\"T_364bd_row10_col2\" class=\"data row10 col2\" >Hong Kong</td>\n",
+ " <td id=\"T_364bd_row10_col3\" class=\"data row10 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_070f8_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_070f8_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_070f8_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_070f8_row11_col2\" class=\"data row11 col2\" >Venus</td>\n",
- " <td id=\"T_070f8_row11_col3\" class=\"data row11 col3\" >❌</td>\n",
+ " <th id=\"T_364bd_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
+ " <td id=\"T_364bd_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
+ " <td id=\"T_364bd_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
+ " <td id=\"T_364bd_row11_col2\" class=\"data row11 col2\" >Venus</td>\n",
+ " <td id=\"T_364bd_row11_col3\" class=\"data row11 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_070f8_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_070f8_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_070f8_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_070f8_row12_col2\" class=\"data row12 col2\" >1457</td>\n",
- " <td id=\"T_070f8_row12_col3\" class=\"data row12 col3\" >❌</td>\n",
+ " <th id=\"T_364bd_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
+ " <td id=\"T_364bd_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
+ " <td id=\"T_364bd_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
+ " <td id=\"T_364bd_row12_col2\" class=\"data row12 col2\" >1457</td>\n",
+ " <td id=\"T_364bd_row12_col3\" class=\"data row12 col3\" >❌</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n"
],
"text/plain": [
- "<pandas.io.formats.style.Styler at 0x7fa029297fd0>"
+ "<pandas.io.formats.style.Styler at 0x7f2d505339a0>"
]
},
"metadata": {},
@@ -433,7 +593,7 @@
},
{
"cell_type": "code",
- "execution_count": 9,
+ "execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
@@ -459,7 +619,7 @@
},
{
"cell_type": "code",
- "execution_count": 10,
+ "execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
@@ -483,7 +643,7 @@
},
{
"cell_type": "code",
- "execution_count": 11,
+ "execution_count": 12,
"metadata": {},
"outputs": [
{
@@ -546,7 +706,7 @@
"('E. L. Doctorow', None)"
]
},
- "execution_count": 11,
+ "execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
@@ -565,14 +725,14 @@
},
{
"cell_type": "code",
- "execution_count": 12,
+ "execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
- "100%|██████████| 13/13 [00:00<00:00, 244.05it/s]"
+ "100%|██████████| 13/13 [00:00<00:00, 208.59it/s]"
]
},
{
@@ -593,120 +753,120 @@
"data": {
"text/html": [
"<style type=\"text/css\">\n",
- "#T_51a84_ th {\n",
+ "#T_5ecf5 th {\n",
" text-align: left;\n",
"}\n",
- "#T_51a84_ td {\n",
+ "#T_5ecf5 td {\n",
" text-align: left;\n",
"}\n",
"</style>\n",
- "<table id=\"T_51a84_\">\n",
+ "<table id=\"T_5ecf5\">\n",
" <thead>\n",
" <tr>\n",
" <th class=\"blank level0\" >&nbsp;</th>\n",
- " <th class=\"col_heading level0 col0\" >question</th>\n",
- " <th class=\"col_heading level0 col1\" >answer</th>\n",
- " <th class=\"col_heading level0 col2\" >prediction</th>\n",
- " <th class=\"col_heading level0 col3\" >correct</th>\n",
+ " <th id=\"T_5ecf5_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
+ " <th id=\"T_5ecf5_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
+ " <th id=\"T_5ecf5_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
+ " <th id=\"T_5ecf5_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
- " <th id=\"T_51a84_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_51a84_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_51a84_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_51a84_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_51a84_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
+ " <th id=\"T_5ecf5_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
+ " <td id=\"T_5ecf5_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
+ " <td id=\"T_5ecf5_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
+ " <td id=\"T_5ecf5_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
+ " <td id=\"T_5ecf5_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_51a84_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_51a84_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_51a84_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_51a84_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
- " <td id=\"T_51a84_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
+ " <th id=\"T_5ecf5_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
+ " <td id=\"T_5ecf5_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
+ " <td id=\"T_5ecf5_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
+ " <td id=\"T_5ecf5_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
+ " <td id=\"T_5ecf5_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_51a84_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_51a84_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_51a84_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_51a84_row2_col2\" class=\"data row2 col2\" >Daytona Beach, Florida</td>\n",
- " <td id=\"T_51a84_row2_col3\" class=\"data row2 col3\" >❌</td>\n",
+ " <th id=\"T_5ecf5_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
+ " <td id=\"T_5ecf5_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
+ " <td id=\"T_5ecf5_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
+ " <td id=\"T_5ecf5_row2_col2\" class=\"data row2 col2\" >Daytona Beach, Florida</td>\n",
+ " <td id=\"T_5ecf5_row2_col3\" class=\"data row2 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_51a84_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_51a84_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_51a84_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_51a84_row3_col2\" class=\"data row3 col2\" >1856</td>\n",
- " <td id=\"T_51a84_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
+ " <th id=\"T_5ecf5_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
+ " <td id=\"T_5ecf5_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
+ " <td id=\"T_5ecf5_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
+ " <td id=\"T_5ecf5_row3_col2\" class=\"data row3 col2\" >1856</td>\n",
+ " <td id=\"T_5ecf5_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_51a84_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_51a84_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_51a84_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_51a84_row4_col2\" class=\"data row4 col2\" >Studs Terkel</td>\n",
- " <td id=\"T_51a84_row4_col3\" class=\"data row4 col3\" >❌</td>\n",
+ " <th id=\"T_5ecf5_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
+ " <td id=\"T_5ecf5_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
+ " <td id=\"T_5ecf5_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
+ " <td id=\"T_5ecf5_row4_col2\" class=\"data row4 col2\" >Studs Terkel</td>\n",
+ " <td id=\"T_5ecf5_row4_col3\" class=\"data row4 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_51a84_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_51a84_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_51a84_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_51a84_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_51a84_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
+ " <th id=\"T_5ecf5_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
+ " <td id=\"T_5ecf5_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
+ " <td id=\"T_5ecf5_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
+ " <td id=\"T_5ecf5_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
+ " <td id=\"T_5ecf5_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_51a84_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_51a84_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_51a84_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_51a84_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
- " <td id=\"T_51a84_row6_col3\" class=\"data row6 col3\" >❌</td>\n",
+ " <th id=\"T_5ecf5_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
+ " <td id=\"T_5ecf5_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
+ " <td id=\"T_5ecf5_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
+ " <td id=\"T_5ecf5_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
+ " <td id=\"T_5ecf5_row6_col3\" class=\"data row6 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_51a84_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_51a84_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_51a84_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_51a84_row7_col2\" class=\"data row7 col2\" >5 August 1862</td>\n",
- " <td id=\"T_51a84_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
+ " <th id=\"T_5ecf5_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
+ " <td id=\"T_5ecf5_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
+ " <td id=\"T_5ecf5_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
+ " <td id=\"T_5ecf5_row7_col2\" class=\"data row7 col2\" >5 August 1862</td>\n",
+ " <td id=\"T_5ecf5_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_51a84_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_51a84_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_51a84_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_51a84_row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
- " <td id=\"T_51a84_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
+ " <th id=\"T_5ecf5_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
+ " <td id=\"T_5ecf5_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
+ " <td id=\"T_5ecf5_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
+ " <td id=\"T_5ecf5_row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
+ " <td id=\"T_5ecf5_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_51a84_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_51a84_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_51a84_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_51a84_row9_col2\" class=\"data row9 col2\" >Jet Propulsion</td>\n",
- " <td id=\"T_51a84_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
+ " <th id=\"T_5ecf5_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
+ " <td id=\"T_5ecf5_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
+ " <td id=\"T_5ecf5_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
+ " <td id=\"T_5ecf5_row9_col2\" class=\"data row9 col2\" >Jet Propulsion</td>\n",
+ " <td id=\"T_5ecf5_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_51a84_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_51a84_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_51a84_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_51a84_row10_col2\" class=\"data row10 col2\" >Los Angeles, California</td>\n",
- " <td id=\"T_51a84_row10_col3\" class=\"data row10 col3\" >❌</td>\n",
+ " <th id=\"T_5ecf5_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
+ " <td id=\"T_5ecf5_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
+ " <td id=\"T_5ecf5_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
+ " <td id=\"T_5ecf5_row10_col2\" class=\"data row10 col2\" >Los Angeles, California</td>\n",
+ " <td id=\"T_5ecf5_row10_col3\" class=\"data row10 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_51a84_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_51a84_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_51a84_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_51a84_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
- " <td id=\"T_51a84_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
+ " <th id=\"T_5ecf5_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
+ " <td id=\"T_5ecf5_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
+ " <td id=\"T_5ecf5_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
+ " <td id=\"T_5ecf5_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
+ " <td id=\"T_5ecf5_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_51a84_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_51a84_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_51a84_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_51a84_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
- " <td id=\"T_51a84_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
+ " <th id=\"T_5ecf5_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
+ " <td id=\"T_5ecf5_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
+ " <td id=\"T_5ecf5_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
+ " <td id=\"T_5ecf5_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
+ " <td id=\"T_5ecf5_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n"
],
"text/plain": [
- "<pandas.io.formats.style.Styler at 0x7fa01eef6d90>"
+ "<pandas.io.formats.style.Styler at 0x7f2d5051aee0>"
]
},
"metadata": {},
@@ -737,7 +897,7 @@
},
{
"cell_type": "code",
- "execution_count": 13,
+ "execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
@@ -766,7 +926,7 @@
},
{
"cell_type": "code",
- "execution_count": 14,
+ "execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
@@ -798,7 +958,7 @@
},
{
"cell_type": "code",
- "execution_count": 15,
+ "execution_count": 16,
"metadata": {},
"outputs": [
{
@@ -869,7 +1029,7 @@
"('Ocala, Florida', None)"
]
},
- "execution_count": 15,
+ "execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
@@ -888,14 +1048,14 @@
},
{
"cell_type": "code",
- "execution_count": 16,
+ "execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
- "100%|██████████| 13/13 [00:00<00:00, 18.90it/s]"
+ "100%|██████████| 13/13 [00:00<00:00, 16.43it/s]"
]
},
{
@@ -916,120 +1076,120 @@
"data": {
"text/html": [
"<style type=\"text/css\">\n",
- "#T_2cf39_ th {\n",
+ "#T_3695c th {\n",
" text-align: left;\n",
"}\n",
- "#T_2cf39_ td {\n",
+ "#T_3695c td {\n",
" text-align: left;\n",
"}\n",
"</style>\n",
- "<table id=\"T_2cf39_\">\n",
+ "<table id=\"T_3695c\">\n",
" <thead>\n",
" <tr>\n",
" <th class=\"blank level0\" >&nbsp;</th>\n",
- " <th class=\"col_heading level0 col0\" >question</th>\n",
- " <th class=\"col_heading level0 col1\" >answer</th>\n",
- " <th class=\"col_heading level0 col2\" >prediction</th>\n",
- " <th class=\"col_heading level0 col3\" >correct</th>\n",
+ " <th id=\"T_3695c_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
+ " <th id=\"T_3695c_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
+ " <th id=\"T_3695c_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
+ " <th id=\"T_3695c_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
- " <th id=\"T_2cf39_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_2cf39_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_2cf39_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_2cf39_row0_col2\" class=\"data row0 col2\" >Julia Peterkin</td>\n",
- " <td id=\"T_2cf39_row0_col3\" class=\"data row0 col3\" >❌</td>\n",
+ " <th id=\"T_3695c_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
+ " <td id=\"T_3695c_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
+ " <td id=\"T_3695c_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
+ " <td id=\"T_3695c_row0_col2\" class=\"data row0 col2\" >Julia Peterkin</td>\n",
+ " <td id=\"T_3695c_row0_col3\" class=\"data row0 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_2cf39_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_2cf39_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_2cf39_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_2cf39_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
- " <td id=\"T_2cf39_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
+ " <th id=\"T_3695c_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
+ " <td id=\"T_3695c_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
+ " <td id=\"T_3695c_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
+ " <td id=\"T_3695c_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
+ " <td id=\"T_3695c_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_2cf39_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_2cf39_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_2cf39_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_2cf39_row2_col2\" class=\"data row2 col2\" >Ocala, Florida</td>\n",
- " <td id=\"T_2cf39_row2_col3\" class=\"data row2 col3\" >❌</td>\n",
+ " <th id=\"T_3695c_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
+ " <td id=\"T_3695c_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
+ " <td id=\"T_3695c_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
+ " <td id=\"T_3695c_row2_col2\" class=\"data row2 col2\" >Ocala, Florida</td>\n",
+ " <td id=\"T_3695c_row2_col3\" class=\"data row2 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_2cf39_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_2cf39_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_2cf39_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_2cf39_row3_col2\" class=\"data row3 col2\" >1824</td>\n",
- " <td id=\"T_2cf39_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
+ " <th id=\"T_3695c_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
+ " <td id=\"T_3695c_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
+ " <td id=\"T_3695c_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
+ " <td id=\"T_3695c_row3_col2\" class=\"data row3 col2\" >1824</td>\n",
+ " <td id=\"T_3695c_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_2cf39_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_2cf39_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_2cf39_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_2cf39_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
- " <td id=\"T_2cf39_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
+ " <th id=\"T_3695c_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
+ " <td id=\"T_3695c_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
+ " <td id=\"T_3695c_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
+ " <td id=\"T_3695c_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
+ " <td id=\"T_3695c_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_2cf39_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_2cf39_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_2cf39_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_2cf39_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_2cf39_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
+ " <th id=\"T_3695c_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
+ " <td id=\"T_3695c_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
+ " <td id=\"T_3695c_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
+ " <td id=\"T_3695c_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
+ " <td id=\"T_3695c_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_2cf39_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_2cf39_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_2cf39_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_2cf39_row6_col2\" class=\"data row6 col2\" >Cheshire, UK</td>\n",
- " <td id=\"T_2cf39_row6_col3\" class=\"data row6 col3\" >✔️</td>\n",
+ " <th id=\"T_3695c_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
+ " <td id=\"T_3695c_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
+ " <td id=\"T_3695c_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
+ " <td id=\"T_3695c_row6_col2\" class=\"data row6 col2\" >Cheshire, UK</td>\n",
+ " <td id=\"T_3695c_row6_col3\" class=\"data row6 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_2cf39_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_2cf39_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_2cf39_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_2cf39_row7_col2\" class=\"data row7 col2\" >September 27, 1974</td>\n",
- " <td id=\"T_2cf39_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
+ " <th id=\"T_3695c_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
+ " <td id=\"T_3695c_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
+ " <td id=\"T_3695c_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
+ " <td id=\"T_3695c_row7_col2\" class=\"data row7 col2\" >September 27, 1974</td>\n",
+ " <td id=\"T_3695c_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_2cf39_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_2cf39_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_2cf39_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_2cf39_row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
- " <td id=\"T_2cf39_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
+ " <th id=\"T_3695c_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
+ " <td id=\"T_3695c_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
+ " <td id=\"T_3695c_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
+ " <td id=\"T_3695c_row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
+ " <td id=\"T_3695c_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_2cf39_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_2cf39_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_2cf39_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_2cf39_row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",
- " <td id=\"T_2cf39_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
+ " <th id=\"T_3695c_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
+ " <td id=\"T_3695c_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
+ " <td id=\"T_3695c_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
+ " <td id=\"T_3695c_row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",
+ " <td id=\"T_3695c_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_2cf39_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_2cf39_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_2cf39_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_2cf39_row10_col2\" class=\"data row10 col2\" >Italy</td>\n",
- " <td id=\"T_2cf39_row10_col3\" class=\"data row10 col3\" >❌</td>\n",
+ " <th id=\"T_3695c_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
+ " <td id=\"T_3695c_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
+ " <td id=\"T_3695c_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
+ " <td id=\"T_3695c_row10_col2\" class=\"data row10 col2\" >Italy</td>\n",
+ " <td id=\"T_3695c_row10_col3\" class=\"data row10 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_2cf39_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_2cf39_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_2cf39_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_2cf39_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
- " <td id=\"T_2cf39_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
+ " <th id=\"T_3695c_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
+ " <td id=\"T_3695c_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
+ " <td id=\"T_3695c_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
+ " <td id=\"T_3695c_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
+ " <td id=\"T_3695c_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_2cf39_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_2cf39_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_2cf39_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_2cf39_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
- " <td id=\"T_2cf39_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
+ " <th id=\"T_3695c_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
+ " <td id=\"T_3695c_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
+ " <td id=\"T_3695c_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
+ " <td id=\"T_3695c_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
+ " <td id=\"T_3695c_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n"
],
"text/plain": [
- "<pandas.io.formats.style.Styler at 0x7fa01e93e3d0>"
+ "<pandas.io.formats.style.Styler at 0x7f2c97dcf2b0>"
]
},
"metadata": {},
@@ -1064,7 +1224,7 @@
},
{
"cell_type": "code",
- "execution_count": 17,
+ "execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
@@ -1094,7 +1254,7 @@
},
{
"cell_type": "code",
- "execution_count": 18,
+ "execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
@@ -1123,7 +1283,7 @@
},
{
"cell_type": "code",
- "execution_count": 19,
+ "execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
@@ -1167,7 +1327,7 @@
},
{
"cell_type": "code",
- "execution_count": 20,
+ "execution_count": 21,
"metadata": {},
"outputs": [
{
@@ -1292,7 +1452,7 @@
"('Gainesville, Florida', None)"
]
},
- "execution_count": 20,
+ "execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
@@ -1311,14 +1471,14 @@
},
{
"cell_type": "code",
- "execution_count": 21,
+ "execution_count": 22,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
- "100%|██████████| 13/13 [00:00<00:00, 89.68it/s]"
+ "100%|██████████| 13/13 [00:00<00:00, 83.91it/s]"
]
},
{
@@ -1339,120 +1499,120 @@
"data": {
"text/html": [
"<style type=\"text/css\">\n",
- "#T_f09c3_ th {\n",
+ "#T_83eb7 th {\n",
" text-align: left;\n",
"}\n",
- "#T_f09c3_ td {\n",
+ "#T_83eb7 td {\n",
" text-align: left;\n",
"}\n",
"</style>\n",
- "<table id=\"T_f09c3_\">\n",
+ "<table id=\"T_83eb7\">\n",
" <thead>\n",
" <tr>\n",
" <th class=\"blank level0\" >&nbsp;</th>\n",
- " <th class=\"col_heading level0 col0\" >question</th>\n",
- " <th class=\"col_heading level0 col1\" >answer</th>\n",
- " <th class=\"col_heading level0 col2\" >prediction</th>\n",
- " <th class=\"col_heading level0 col3\" >correct</th>\n",
+ " <th id=\"T_83eb7_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
+ " <th id=\"T_83eb7_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
+ " <th id=\"T_83eb7_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
+ " <th id=\"T_83eb7_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
- " <th id=\"T_f09c3_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_f09c3_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_f09c3_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_f09c3_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_f09c3_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
+ " <th id=\"T_83eb7_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
+ " <td id=\"T_83eb7_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
+ " <td id=\"T_83eb7_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
+ " <td id=\"T_83eb7_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
+ " <td id=\"T_83eb7_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_f09c3_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_f09c3_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_f09c3_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_f09c3_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
- " <td id=\"T_f09c3_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
+ " <th id=\"T_83eb7_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
+ " <td id=\"T_83eb7_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
+ " <td id=\"T_83eb7_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
+ " <td id=\"T_83eb7_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
+ " <td id=\"T_83eb7_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_f09c3_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_f09c3_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_f09c3_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_f09c3_row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",
- " <td id=\"T_f09c3_row2_col3\" class=\"data row2 col3\" >✔️</td>\n",
+ " <th id=\"T_83eb7_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
+ " <td id=\"T_83eb7_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
+ " <td id=\"T_83eb7_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
+ " <td id=\"T_83eb7_row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",
+ " <td id=\"T_83eb7_row2_col3\" class=\"data row2 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_f09c3_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_f09c3_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_f09c3_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_f09c3_row3_col2\" class=\"data row3 col2\" >1947</td>\n",
- " <td id=\"T_f09c3_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
+ " <th id=\"T_83eb7_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
+ " <td id=\"T_83eb7_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
+ " <td id=\"T_83eb7_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
+ " <td id=\"T_83eb7_row3_col2\" class=\"data row3 col2\" >1947</td>\n",
+ " <td id=\"T_83eb7_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_f09c3_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_f09c3_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_f09c3_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_f09c3_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
- " <td id=\"T_f09c3_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
+ " <th id=\"T_83eb7_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
+ " <td id=\"T_83eb7_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
+ " <td id=\"T_83eb7_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
+ " <td id=\"T_83eb7_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
+ " <td id=\"T_83eb7_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_f09c3_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_f09c3_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_f09c3_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_f09c3_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_f09c3_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
+ " <th id=\"T_83eb7_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
+ " <td id=\"T_83eb7_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
+ " <td id=\"T_83eb7_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
+ " <td id=\"T_83eb7_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
+ " <td id=\"T_83eb7_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_f09c3_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_f09c3_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_f09c3_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_f09c3_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
- " <td id=\"T_f09c3_row6_col3\" class=\"data row6 col3\" >❌</td>\n",
+ " <th id=\"T_83eb7_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
+ " <td id=\"T_83eb7_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
+ " <td id=\"T_83eb7_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
+ " <td id=\"T_83eb7_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
+ " <td id=\"T_83eb7_row6_col3\" class=\"data row6 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_f09c3_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_f09c3_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_f09c3_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_f09c3_row7_col2\" class=\"data row7 col2\" >1965</td>\n",
- " <td id=\"T_f09c3_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
+ " <th id=\"T_83eb7_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
+ " <td id=\"T_83eb7_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
+ " <td id=\"T_83eb7_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
+ " <td id=\"T_83eb7_row7_col2\" class=\"data row7 col2\" >1965</td>\n",
+ " <td id=\"T_83eb7_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_f09c3_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_f09c3_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_f09c3_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_f09c3_row8_col2\" class=\"data row8 col2\" >River Foyle</td>\n",
- " <td id=\"T_f09c3_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
+ " <th id=\"T_83eb7_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
+ " <td id=\"T_83eb7_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
+ " <td id=\"T_83eb7_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
+ " <td id=\"T_83eb7_row8_col2\" class=\"data row8 col2\" >River Foyle</td>\n",
+ " <td id=\"T_83eb7_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_f09c3_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_f09c3_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_f09c3_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_f09c3_row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",
- " <td id=\"T_f09c3_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
+ " <th id=\"T_83eb7_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
+ " <td id=\"T_83eb7_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
+ " <td id=\"T_83eb7_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
+ " <td id=\"T_83eb7_row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",
+ " <td id=\"T_83eb7_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_f09c3_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_f09c3_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_f09c3_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_f09c3_row10_col2\" class=\"data row10 col2\" >Guildford, Surrey</td>\n",
- " <td id=\"T_f09c3_row10_col3\" class=\"data row10 col3\" >✔️</td>\n",
+ " <th id=\"T_83eb7_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
+ " <td id=\"T_83eb7_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
+ " <td id=\"T_83eb7_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
+ " <td id=\"T_83eb7_row10_col2\" class=\"data row10 col2\" >Guildford, Surrey</td>\n",
+ " <td id=\"T_83eb7_row10_col3\" class=\"data row10 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_f09c3_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_f09c3_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_f09c3_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_f09c3_row11_col2\" class=\"data row11 col2\" >Carrington</td>\n",
- " <td id=\"T_f09c3_row11_col3\" class=\"data row11 col3\" >❌</td>\n",
+ " <th id=\"T_83eb7_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
+ " <td id=\"T_83eb7_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
+ " <td id=\"T_83eb7_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
+ " <td id=\"T_83eb7_row11_col2\" class=\"data row11 col2\" >Carrington</td>\n",
+ " <td id=\"T_83eb7_row11_col3\" class=\"data row11 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_f09c3_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_f09c3_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_f09c3_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_f09c3_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
- " <td id=\"T_f09c3_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
+ " <th id=\"T_83eb7_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
+ " <td id=\"T_83eb7_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
+ " <td id=\"T_83eb7_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
+ " <td id=\"T_83eb7_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
+ " <td id=\"T_83eb7_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n"
],
"text/plain": [
- "<pandas.io.formats.style.Styler at 0x7fa01d1be810>"
+ "<pandas.io.formats.style.Styler at 0x7f2d50533ee0>"
]
},
"metadata": {},
@@ -1489,7 +1649,7 @@
},
{
"cell_type": "code",
- "execution_count": 22,
+ "execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
@@ -1526,7 +1686,7 @@
},
{
"cell_type": "code",
- "execution_count": 23,
+ "execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
@@ -1559,7 +1719,7 @@
},
{
"cell_type": "code",
- "execution_count": 24,
+ "execution_count": 25,
"metadata": {},
"outputs": [
{
@@ -1777,7 +1937,7 @@
"('1828', None)"
]
},
- "execution_count": 24,
+ "execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
@@ -1788,14 +1948,14 @@
},
{
"cell_type": "code",
- "execution_count": 25,
+ "execution_count": 26,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
- "100%|██████████| 13/13 [00:01<00:00, 8.38it/s]"
+ "100%|██████████| 13/13 [00:01<00:00, 8.50it/s]"
]
},
{
@@ -1816,120 +1976,120 @@
"data": {
"text/html": [
"<style type=\"text/css\">\n",
- "#T_b8b4e_ th {\n",
+ "#T_b6174 th {\n",
" text-align: left;\n",
"}\n",
- "#T_b8b4e_ td {\n",
+ "#T_b6174 td {\n",
" text-align: left;\n",
"}\n",
"</style>\n",
- "<table id=\"T_b8b4e_\">\n",
+ "<table id=\"T_b6174\">\n",
" <thead>\n",
" <tr>\n",
" <th class=\"blank level0\" >&nbsp;</th>\n",
- " <th class=\"col_heading level0 col0\" >question</th>\n",
- " <th class=\"col_heading level0 col1\" >answer</th>\n",
- " <th class=\"col_heading level0 col2\" >prediction</th>\n",
- " <th class=\"col_heading level0 col3\" >correct</th>\n",
+ " <th id=\"T_b6174_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
+ " <th id=\"T_b6174_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
+ " <th id=\"T_b6174_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
+ " <th id=\"T_b6174_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
- " <th id=\"T_b8b4e_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_b8b4e_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_b8b4e_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_b8b4e_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_b8b4e_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
+ " <th id=\"T_b6174_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
+ " <td id=\"T_b6174_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
+ " <td id=\"T_b6174_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
+ " <td id=\"T_b6174_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
+ " <td id=\"T_b6174_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_b8b4e_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_b8b4e_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_b8b4e_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_b8b4e_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
- " <td id=\"T_b8b4e_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
+ " <th id=\"T_b6174_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
+ " <td id=\"T_b6174_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
+ " <td id=\"T_b6174_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
+ " <td id=\"T_b6174_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
+ " <td id=\"T_b6174_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_b8b4e_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_b8b4e_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_b8b4e_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_b8b4e_row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",
- " <td id=\"T_b8b4e_row2_col3\" class=\"data row2 col3\" >✔️</td>\n",
+ " <th id=\"T_b6174_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
+ " <td id=\"T_b6174_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
+ " <td id=\"T_b6174_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
+ " <td id=\"T_b6174_row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",
+ " <td id=\"T_b6174_row2_col3\" class=\"data row2 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_b8b4e_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_b8b4e_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_b8b4e_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_b8b4e_row3_col2\" class=\"data row3 col2\" >1828</td>\n",
- " <td id=\"T_b8b4e_row3_col3\" class=\"data row3 col3\" >✔️</td>\n",
+ " <th id=\"T_b6174_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
+ " <td id=\"T_b6174_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
+ " <td id=\"T_b6174_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
+ " <td id=\"T_b6174_row3_col2\" class=\"data row3 col2\" >1828</td>\n",
+ " <td id=\"T_b6174_row3_col3\" class=\"data row3 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_b8b4e_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_b8b4e_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_b8b4e_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_b8b4e_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
- " <td id=\"T_b8b4e_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
+ " <th id=\"T_b6174_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
+ " <td id=\"T_b6174_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
+ " <td id=\"T_b6174_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
+ " <td id=\"T_b6174_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
+ " <td id=\"T_b6174_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_b8b4e_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_b8b4e_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_b8b4e_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_b8b4e_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_b8b4e_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
+ " <th id=\"T_b6174_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
+ " <td id=\"T_b6174_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
+ " <td id=\"T_b6174_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
+ " <td id=\"T_b6174_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
+ " <td id=\"T_b6174_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_b8b4e_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_b8b4e_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_b8b4e_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_b8b4e_row6_col2\" class=\"data row6 col2\" >Cheshire</td>\n",
- " <td id=\"T_b8b4e_row6_col3\" class=\"data row6 col3\" >✔️</td>\n",
+ " <th id=\"T_b6174_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
+ " <td id=\"T_b6174_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
+ " <td id=\"T_b6174_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
+ " <td id=\"T_b6174_row6_col2\" class=\"data row6 col2\" >Cheshire</td>\n",
+ " <td id=\"T_b6174_row6_col3\" class=\"data row6 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_b8b4e_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_b8b4e_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_b8b4e_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_b8b4e_row7_col2\" class=\"data row7 col2\" >1958</td>\n",
- " <td id=\"T_b8b4e_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
+ " <th id=\"T_b6174_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
+ " <td id=\"T_b6174_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
+ " <td id=\"T_b6174_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
+ " <td id=\"T_b6174_row7_col2\" class=\"data row7 col2\" >1958</td>\n",
+ " <td id=\"T_b6174_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_b8b4e_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_b8b4e_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_b8b4e_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_b8b4e_row8_col2\" class=\"data row8 col2\" >Castletown River</td>\n",
- " <td id=\"T_b8b4e_row8_col3\" class=\"data row8 col3\" >✔️</td>\n",
+ " <th id=\"T_b6174_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
+ " <td id=\"T_b6174_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
+ " <td id=\"T_b6174_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
+ " <td id=\"T_b6174_row8_col2\" class=\"data row8 col2\" >Castletown River</td>\n",
+ " <td id=\"T_b6174_row8_col3\" class=\"data row8 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_b8b4e_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_b8b4e_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_b8b4e_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_b8b4e_row9_col2\" class=\"data row9 col2\" >Aeronautics</td>\n",
- " <td id=\"T_b8b4e_row9_col3\" class=\"data row9 col3\" >✔️</td>\n",
+ " <th id=\"T_b6174_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
+ " <td id=\"T_b6174_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
+ " <td id=\"T_b6174_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
+ " <td id=\"T_b6174_row9_col2\" class=\"data row9 col2\" >Aeronautics</td>\n",
+ " <td id=\"T_b6174_row9_col3\" class=\"data row9 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_b8b4e_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_b8b4e_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_b8b4e_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_b8b4e_row10_col2\" class=\"data row10 col2\" >Surrey</td>\n",
- " <td id=\"T_b8b4e_row10_col3\" class=\"data row10 col3\" >✔️</td>\n",
+ " <th id=\"T_b6174_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
+ " <td id=\"T_b6174_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
+ " <td id=\"T_b6174_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
+ " <td id=\"T_b6174_row10_col2\" class=\"data row10 col2\" >Surrey</td>\n",
+ " <td id=\"T_b6174_row10_col3\" class=\"data row10 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_b8b4e_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_b8b4e_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_b8b4e_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_b8b4e_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
- " <td id=\"T_b8b4e_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
+ " <th id=\"T_b6174_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
+ " <td id=\"T_b6174_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
+ " <td id=\"T_b6174_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
+ " <td id=\"T_b6174_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
+ " <td id=\"T_b6174_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
" </tr>\n",
" <tr>\n",
- " <th id=\"T_b8b4e_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_b8b4e_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_b8b4e_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_b8b4e_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
- " <td id=\"T_b8b4e_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
+ " <th id=\"T_b6174_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
+ " <td id=\"T_b6174_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
+ " <td id=\"T_b6174_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
+ " <td id=\"T_b6174_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
+ " <td id=\"T_b6174_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n"
],
"text/plain": [
- "<pandas.io.formats.style.Styler at 0x7fa01d1dbb50>"
+ "<pandas.io.formats.style.Styler at 0x7f2c9404ee80>"
]
},
"metadata": {},
@@ -1945,15 +2105,39 @@
"cell_type": "markdown",
"metadata": {},
"source": [
+ "### Additional Programs...\n",
+ "\n",
"While Program 5 is undeniably better than, say, Program 1, it's far from perfect.\n",
"\n",
- "Many different ideas can easily be implemented to make it more general or more powerful. Refer to the DSP paper for inspiration!"
+ "Many different ideas can easily be implemented to make it more general or more powerful. Refer to the DSP paper for inspiration!\n",
+ "\n",
+ "One of your first improvements to Programs 4 and 5 would be to allow it to do more than two hops and to automatically stop when additional hops are not required.\n",
+ "\n",
+ "**Hint:** This is pretty easy to do. You can start with Program 4 and modify `hop_template` so its `SearchQuery` produces the string `N/A` when additional hops are not required.\n",
+ "\n",
+ "In the `multihop_search_v1` function, you can then break from the loop — i.e., `if completions.query == 'N/A': break` whenever this happens. How will you apply this to Program 5?\n",
+ "\n",
+ "```\n",
+ "SearchQueryWithStopping = dsp.Type(\n",
+ " prefix=\"Search Query:\",\n",
+ " desc=\"${a simple question for seeking the missing or required information --- say N/A if the context above contains all of the required information}\"\n",
+ ")\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 27,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Program 4 with max_hops=4 and automatic stopping."
]
}
],
"metadata": {
"kernelspec": {
- "display_name": "colbert-api-001",
+ "display_name": "py38",
"language": "python",
"name": "python3"
},
@@ -1967,12 +2151,12 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.7.11"
+ "version": "3.8.16"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
- "hash": "08fdf29897c5b8f3cfe29625f379bb8bd67a7135431cc64e36a1afcc33cadfae"
+ "hash": "ff8889dd26469456638524e92174013821322a2ab8e175954d0360930982a899"
}
}
},