DSPy: The framework for programming—not prompting—language models
Python
Update notebook to work on Colab
Omar Khattabf7d2904b9fc9
diff --git a/intro.ipynb b/intro.ipynbindex 294f2acee..c60a69708 100644--- a/intro.ipynb+++ b/intro.ipynb@@ -5,7 +5,7 @@"cell_type": "markdown","metadata": {},"source": [- "## 🤝🎓 𝗗𝗦𝗣: 𝗗EMONSTRATE–𝗦EARCH–𝗣REDICT\n",+ "## 🎓 𝗗𝗦𝗣: 𝗗EMONSTRATE–𝗦EARCH–𝗣REDICT\n","\n","This notebook illustrates the Demonstrate–Search–Predict (𝗗𝗦𝗣) framework on the multi-hop question answering (QA) task.\n","\n",@@ -17,15 +17,9 @@"cell_type": "markdown","metadata": {},"source": [- "### Setting Up\n",- "\n",- "We'll start by setting up the language model (LM) and retrieval model (RM).\n",- "\n",- "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",- "\n",- "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",+ "### Installation\n","\n",- "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"+ "If you haven't installed **DSP** already, let's do that."]},{@@ -37,12 +31,171 @@"name": "stdout","output_type": "stream","text": [+ "Requirement already satisfied: dsp-ml in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (0.1.1)\n",+ "Requirement already satisfied: spacy in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (3.5.0)\n",+ "Requirement already satisfied: backoff in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (2.2.1)\n",+ "Requirement already satisfied: jupyter in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (1.0.0)\n",+ "Requirement already satisfied: pandas in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (1.5.3)\n",+ "Requirement already satisfied: joblib in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (1.2.0)\n",+ "Requirement already satisfied: openai in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (0.26.1)\n",+ "Requirement already satisfied: regex in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (2022.10.31)\n",+ "Requirement already satisfied: ipywidgets in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter->dsp-ml) (8.0.4)\n",+ "Requirement already satisfied: notebook in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter->dsp-ml) (6.5.2)\n",+ "Requirement already satisfied: ipykernel in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter->dsp-ml) (6.20.2)\n",+ "Requirement already satisfied: jupyter-console in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter->dsp-ml) (6.4.4)\n",+ "Requirement already satisfied: qtconsole in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter->dsp-ml) (5.4.0)\n",+ "Requirement already satisfied: nbconvert in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter->dsp-ml) (7.2.8)\n",+ "Requirement already satisfied: requests>=2.20 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from openai->dsp-ml) (2.28.2)\n",+ "Requirement already satisfied: tqdm in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from openai->dsp-ml) (4.64.1)\n",+ "Requirement already satisfied: aiohttp in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from openai->dsp-ml) (3.8.3)\n",+ "Requirement already satisfied: pytz>=2020.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from pandas->dsp-ml) (2022.7.1)\n",+ "Requirement already satisfied: numpy>=1.20.3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from pandas->dsp-ml) (1.24.1)\n",+ "Requirement already satisfied: python-dateutil>=2.8.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from pandas->dsp-ml) (2.8.2)\n",+ "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (2.4.5)\n",+ "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (3.0.11)\n",+ "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (1.10.4)\n",+ "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (2.0.8)\n",+ "Requirement already satisfied: thinc<8.2.0,>=8.1.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (8.1.7)\n",+ "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (1.0.9)\n",+ "Requirement already satisfied: setuptools in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (65.6.3)\n",+ "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (6.3.0)\n",+ "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (2.0.7)\n",+ "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (1.0.4)\n",+ "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (3.3.0)\n",+ "Requirement already satisfied: packaging>=20.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (23.0)\n",+ "Requirement already satisfied: typer<0.8.0,>=0.3.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (0.7.0)\n",+ "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (3.0.8)\n",+ "Requirement already satisfied: jinja2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (3.1.2)\n",+ "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (1.1.1)\n",+ "Requirement already satisfied: pathy>=0.10.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (0.10.1)\n",+ "Requirement already satisfied: typing-extensions>=4.2.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4->spacy->dsp-ml) (4.4.0)\n",+ "Requirement already satisfied: six>=1.5 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from python-dateutil>=2.8.1->pandas->dsp-ml) (1.16.0)\n",+ "Requirement already satisfied: idna<4,>=2.5 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from requests>=2.20->openai->dsp-ml) (3.4)\n",+ "Requirement already satisfied: charset-normalizer<4,>=2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from requests>=2.20->openai->dsp-ml) (2.1.1)\n",+ "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from requests>=2.20->openai->dsp-ml) (1.26.14)\n",+ "Requirement already satisfied: certifi>=2017.4.17 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from requests>=2.20->openai->dsp-ml) (2022.12.7)\n",+ "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from thinc<8.2.0,>=8.1.0->spacy->dsp-ml) (0.7.9)\n",+ "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from thinc<8.2.0,>=8.1.0->spacy->dsp-ml) (0.0.4)\n",+ "Requirement already satisfied: click<9.0.0,>=7.1.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from typer<0.8.0,>=0.3.0->spacy->dsp-ml) (8.1.3)\n",+ "Requirement already satisfied: frozenlist>=1.1.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from aiohttp->openai->dsp-ml) (1.3.3)\n",+ "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from aiohttp->openai->dsp-ml) (4.0.2)\n",+ "Requirement already satisfied: aiosignal>=1.1.2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from aiohttp->openai->dsp-ml) (1.3.1)\n",+ "Requirement already satisfied: attrs>=17.3.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from aiohttp->openai->dsp-ml) (22.2.0)\n",+ "Requirement already satisfied: yarl<2.0,>=1.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from aiohttp->openai->dsp-ml) (1.8.2)\n",+ "Requirement already satisfied: multidict<7.0,>=4.5 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from aiohttp->openai->dsp-ml) (6.0.4)\n",+ "Requirement already satisfied: jupyter-client>=6.1.12 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (7.4.9)\n",+ "Requirement already satisfied: pyzmq>=17 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (25.0.0)\n",+ "Requirement already satisfied: tornado>=6.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (6.2)\n",+ "Requirement already satisfied: nest-asyncio in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (1.5.6)\n",+ "Requirement already satisfied: debugpy>=1.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (1.6.5)\n",+ "Requirement already satisfied: traitlets>=5.4.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (5.8.1)\n",+ "Requirement already satisfied: ipython>=7.23.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (8.8.0)\n",+ "Requirement already satisfied: comm>=0.1.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (0.1.2)\n",+ "Requirement already satisfied: psutil in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (5.9.4)\n",+ "Requirement already satisfied: matplotlib-inline>=0.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (0.1.6)\n",+ "Requirement already satisfied: jupyterlab-widgets~=3.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipywidgets->jupyter->dsp-ml) (3.0.5)\n",+ "Requirement already satisfied: widgetsnbextension~=4.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipywidgets->jupyter->dsp-ml) (4.0.5)\n",+ "Requirement already satisfied: MarkupSafe>=2.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jinja2->spacy->dsp-ml) (2.1.2)\n",+ "Requirement already satisfied: pygments in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-console->jupyter->dsp-ml) (2.14.0)\n",+ "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-console->jupyter->dsp-ml) (3.0.36)\n",+ "Requirement already satisfied: bleach in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (5.0.1)\n",+ "Requirement already satisfied: nbclient>=0.5.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (0.7.2)\n",+ "Requirement already satisfied: mistune<3,>=2.0.3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (2.0.4)\n",+ "Requirement already satisfied: defusedxml in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (0.7.1)\n",+ "Requirement already satisfied: nbformat>=5.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (5.7.3)\n",+ "Requirement already satisfied: importlib-metadata>=3.6 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (6.0.0)\n",+ "Requirement already satisfied: jupyter-core>=4.7 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (5.1.3)\n",+ "Requirement already satisfied: pandocfilters>=1.4.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (1.5.0)\n",+ "Requirement already satisfied: jupyterlab-pygments in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (0.2.2)\n",+ "Requirement already satisfied: beautifulsoup4 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (4.11.1)\n",+ "Requirement already satisfied: tinycss2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (1.2.1)\n",+ "Requirement already satisfied: Send2Trash>=1.8.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from notebook->jupyter->dsp-ml) (1.8.0)\n",+ "Requirement already satisfied: prometheus-client in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from notebook->jupyter->dsp-ml) (0.15.0)\n",+ "Requirement already satisfied: terminado>=0.8.3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from notebook->jupyter->dsp-ml) (0.17.1)\n",+ "Requirement already satisfied: nbclassic>=0.4.7 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from notebook->jupyter->dsp-ml) (0.4.8)\n",+ "Requirement already satisfied: argon2-cffi in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from notebook->jupyter->dsp-ml) (21.3.0)\n",+ "Requirement already satisfied: ipython-genutils in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from notebook->jupyter->dsp-ml) (0.2.0)\n",+ "Requirement already satisfied: qtpy>=2.0.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from qtconsole->jupyter->dsp-ml) (2.3.0)\n",+ "Requirement already satisfied: zipp>=0.5 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from importlib-metadata>=3.6->nbconvert->jupyter->dsp-ml) (3.11.0)\n",+ "Requirement already satisfied: backcall in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (0.2.0)\n",+ "Requirement already satisfied: pexpect>4.3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (4.8.0)\n",+ "Requirement already satisfied: pickleshare in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (0.7.5)\n",+ "Requirement already satisfied: stack-data in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (0.6.2)\n",+ "Requirement already satisfied: decorator in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (5.1.1)\n",+ "Requirement already satisfied: jedi>=0.16 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (0.18.2)\n",+ "Requirement already satisfied: entrypoints in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-client>=6.1.12->ipykernel->jupyter->dsp-ml) (0.4)\n",+ "Requirement already satisfied: platformdirs>=2.5 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-core>=4.7->nbconvert->jupyter->dsp-ml) (2.6.2)\n",+ "Requirement already satisfied: jupyter-server>=1.8 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (2.1.0)\n",+ "Requirement already satisfied: notebook-shim>=0.1.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (0.2.2)\n",+ "Requirement already satisfied: jsonschema>=2.6 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbformat>=5.1->nbconvert->jupyter->dsp-ml) (4.17.3)\n",+ "Requirement already satisfied: fastjsonschema in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbformat>=5.1->nbconvert->jupyter->dsp-ml) (2.16.2)\n",+ "Requirement already satisfied: wcwidth in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->jupyter-console->jupyter->dsp-ml) (0.2.6)\n",+ "Requirement already satisfied: ptyprocess in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from terminado>=0.8.3->notebook->jupyter->dsp-ml) (0.7.0)\n",+ "Requirement already satisfied: argon2-cffi-bindings in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from argon2-cffi->notebook->jupyter->dsp-ml) (21.2.0)\n",+ "Requirement already satisfied: soupsieve>1.2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from beautifulsoup4->nbconvert->jupyter->dsp-ml) (2.3.2.post1)\n",+ "Requirement already satisfied: webencodings in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from bleach->nbconvert->jupyter->dsp-ml) (0.5.1)\n",+ "Requirement already satisfied: parso<0.9.0,>=0.8.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jedi>=0.16->ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (0.8.3)\n",+ "Requirement already satisfied: importlib-resources>=1.4.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (5.10.2)\n",+ "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (0.19.3)\n",+ "Requirement already satisfied: pkgutil-resolve-name>=1.3.10 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (1.3.10)\n",+ "Requirement already satisfied: websocket-client in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (1.4.2)\n",+ "Requirement already satisfied: jupyter-events>=0.4.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (0.6.3)\n",+ "Requirement already satisfied: anyio<4,>=3.1.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (3.6.2)\n",+ "Requirement already satisfied: jupyter-server-terminals in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (0.4.4)\n",+ "Requirement already satisfied: cffi>=1.0.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from argon2-cffi-bindings->argon2-cffi->notebook->jupyter->dsp-ml) (1.15.1)\n",+ "Requirement already satisfied: asttokens>=2.1.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from stack-data->ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (2.2.1)\n",+ "Requirement already satisfied: pure-eval in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from stack-data->ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (0.2.2)\n",+ "Requirement already satisfied: executing>=1.2.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from stack-data->ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (1.2.0)\n",+ "Requirement already satisfied: sniffio>=1.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from anyio<4,>=3.1.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (1.3.0)\n",+ "Requirement already satisfied: pycparser in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->notebook->jupyter->dsp-ml) (2.21)\n",+ "Requirement already satisfied: rfc3339-validator in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-events>=0.4.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (0.1.4)\n",+ "Requirement already satisfied: python-json-logger>=2.0.4 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-events>=0.4.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (2.0.4)\n",+ "Requirement already satisfied: pyyaml>=5.3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-events>=0.4.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (6.0)\n",+ "Requirement already satisfied: rfc3986-validator>=0.1.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-events>=0.4.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (0.1.1)\n",+ "Requirement already satisfied: fqdn in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (1.5.1)\n",+ "Requirement already satisfied: uri-template in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (1.2.0)\n",+ "Requirement already satisfied: webcolors>=1.11 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (1.12)\n",+ "Requirement already satisfied: isoduration in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (20.11.0)\n",+ "Requirement already satisfied: jsonpointer>1.13 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (2.3)\n",+ "Requirement already satisfied: arrow>=0.15.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from isoduration->jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (1.2.3)\n","env: DSP_NOTEBOOK_CACHEDIR=cache\n"]}],"source": [- "%set_env DSP_NOTEBOOK_CACHEDIR cache\n",+ "!pip install dsp-ml\n",+ "\n",+ "try:\n",+ " import google.colab # When on google Colab, let's clone the notebook so we download the cache.\n",+ " !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",+ " %set_env DSP_NOTEBOOK_CACHEDIR dsp/cache\n",+ "\n",+ "except:\n",+ " %set_env DSP_NOTEBOOK_CACHEDIR cache # Set up the cache for the examples below."+ ]+ },+ {+ "attachments": {},+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Setting Up\n",+ "\n",+ "We'll start by setting up the language model (LM) and retrieval model (RM).\n",+ "\n",+ "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",+ "\n",+ "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",+ "\n",+ "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 2,+ "metadata": {},+ "outputs": [],+ "source": ["%load_ext autoreload\n","%autoreload 2\n","\n",@@ -74,7 +227,7 @@},{"cell_type": "code",- "execution_count": 2,+ "execution_count": 3,"metadata": {},"outputs": [],"source": [@@ -98,7 +251,7 @@},{"cell_type": "code",- "execution_count": 3,+ "execution_count": 4,"metadata": {},"outputs": [],"source": [@@ -135,7 +288,7 @@},{"cell_type": "code",- "execution_count": 4,+ "execution_count": 5,"metadata": {},"outputs": [],"source": [@@ -155,7 +308,7 @@},{"cell_type": "code",- "execution_count": 5,+ "execution_count": 6,"metadata": {},"outputs": [],"source": [@@ -177,7 +330,7 @@},{"cell_type": "code",- "execution_count": 6,+ "execution_count": 7,"metadata": {},"outputs": [{@@ -204,7 +357,7 @@},{"cell_type": "code",- "execution_count": 7,+ "execution_count": 8,"metadata": {},"outputs": [{@@ -268,14 +421,14 @@},{"cell_type": "code",- "execution_count": 8,+ "execution_count": 9,"metadata": {},"outputs": [{"name": "stderr","output_type": "stream","text": [- "100%|██████████| 13/13 [00:00<00:00, 354.31it/s]\n"+ "100%|██████████| 13/13 [00:00<00:00, 384.43it/s]"]},{@@ -285,124 +438,131 @@"Answered 4 / 13 correctly.\n"]},+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "\n"+ ]+ },{"data": {"text/html": ["<style type=\"text/css\">\n",- "#T_070f8_ th {\n",+ "#T_364bd th {\n"," text-align: left;\n","}\n",- "#T_070f8_ td {\n",+ "#T_364bd td {\n"," text-align: left;\n","}\n","</style>\n",- "<table id=\"T_070f8_\">\n",+ "<table id=\"T_364bd\">\n"," <thead>\n"," <tr>\n"," <th class=\"blank level0\" > </th>\n",- " <th class=\"col_heading level0 col0\" >question</th>\n",- " <th class=\"col_heading level0 col1\" >answer</th>\n",- " <th class=\"col_heading level0 col2\" >prediction</th>\n",- " <th class=\"col_heading level0 col3\" >correct</th>\n",+ " <th id=\"T_364bd_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",+ " <th id=\"T_364bd_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",+ " <th id=\"T_364bd_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",+ " <th id=\"T_364bd_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n"," </tr>\n"," </thead>\n"," <tbody>\n"," <tr>\n",- " <th id=\"T_070f8_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",- " <td id=\"T_070f8_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",- " <td id=\"T_070f8_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",- " <td id=\"T_070f8_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",- " <td id=\"T_070f8_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",+ " <th id=\"T_364bd_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",+ " <td id=\"T_364bd_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",+ " <td id=\"T_364bd_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",+ " <td id=\"T_364bd_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",+ " <td id=\"T_364bd_row0_col3\" class=\"data row0 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_070f8_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",- " <td id=\"T_070f8_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",- " <td id=\"T_070f8_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",- " <td id=\"T_070f8_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",- " <td id=\"T_070f8_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",+ " <th id=\"T_364bd_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",+ " <td id=\"T_364bd_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",+ " <td id=\"T_364bd_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",+ " <td id=\"T_364bd_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",+ " <td id=\"T_364bd_row1_col3\" class=\"data row1 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_070f8_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",- " <td id=\"T_070f8_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",- " <td id=\"T_070f8_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",- " <td id=\"T_070f8_row2_col2\" class=\"data row2 col2\" >Melbourne, Australia</td>\n",- " <td id=\"T_070f8_row2_col3\" class=\"data row2 col3\" >❌</td>\n",+ " <th id=\"T_364bd_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",+ " <td id=\"T_364bd_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",+ " <td id=\"T_364bd_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",+ " <td id=\"T_364bd_row2_col2\" class=\"data row2 col2\" >Melbourne, Australia</td>\n",+ " <td id=\"T_364bd_row2_col3\" class=\"data row2 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_070f8_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",- " <td id=\"T_070f8_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",- " <td id=\"T_070f8_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",- " <td id=\"T_070f8_row3_col2\" class=\"data row3 col2\" >1966</td>\n",- " <td id=\"T_070f8_row3_col3\" class=\"data row3 col3\" >❌</td>\n",+ " <th id=\"T_364bd_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",+ " <td id=\"T_364bd_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",+ " <td id=\"T_364bd_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",+ " <td id=\"T_364bd_row3_col2\" class=\"data row3 col2\" >1966</td>\n",+ " <td id=\"T_364bd_row3_col3\" class=\"data row3 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_070f8_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",- " <td id=\"T_070f8_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",- " <td id=\"T_070f8_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",- " <td id=\"T_070f8_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",- " <td id=\"T_070f8_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",+ " <th id=\"T_364bd_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",+ " <td id=\"T_364bd_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",+ " <td id=\"T_364bd_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",+ " <td id=\"T_364bd_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",+ " <td id=\"T_364bd_row4_col3\" class=\"data row4 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_070f8_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",- " <td id=\"T_070f8_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",- " <td id=\"T_070f8_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",- " <td id=\"T_070f8_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",- " <td id=\"T_070f8_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",+ " <th id=\"T_364bd_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",+ " <td id=\"T_364bd_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",+ " <td id=\"T_364bd_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",+ " <td id=\"T_364bd_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",+ " <td id=\"T_364bd_row5_col3\" class=\"data row5 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_070f8_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",- " <td id=\"T_070f8_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",- " <td id=\"T_070f8_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",- " <td id=\"T_070f8_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",- " <td id=\"T_070f8_row6_col3\" class=\"data row6 col3\" >❌</td>\n",+ " <th id=\"T_364bd_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",+ " <td id=\"T_364bd_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",+ " <td id=\"T_364bd_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",+ " <td id=\"T_364bd_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",+ " <td id=\"T_364bd_row6_col3\" class=\"data row6 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_070f8_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",- " <td id=\"T_070f8_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",- " <td id=\"T_070f8_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",- " <td id=\"T_070f8_row7_col2\" class=\"data row7 col2\" >July 3, 1961</td>\n",- " <td id=\"T_070f8_row7_col3\" class=\"data row7 col3\" >❌</td>\n",+ " <th id=\"T_364bd_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",+ " <td id=\"T_364bd_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",+ " <td id=\"T_364bd_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",+ " <td id=\"T_364bd_row7_col2\" class=\"data row7 col2\" >July 3, 1961</td>\n",+ " <td id=\"T_364bd_row7_col3\" class=\"data row7 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_070f8_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",- " <td id=\"T_070f8_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",- " <td id=\"T_070f8_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",- " <td id=\"T_070f8_row8_col2\" class=\"data row8 col2\" >Susquehanna River</td>\n",- " <td id=\"T_070f8_row8_col3\" class=\"data row8 col3\" >❌</td>\n",+ " <th id=\"T_364bd_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",+ " <td id=\"T_364bd_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",+ " <td id=\"T_364bd_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",+ " <td id=\"T_364bd_row8_col2\" class=\"data row8 col2\" >Susquehanna River</td>\n",+ " <td id=\"T_364bd_row8_col3\" class=\"data row8 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_070f8_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",- " <td id=\"T_070f8_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",- " <td id=\"T_070f8_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",- " <td id=\"T_070f8_row9_col2\" class=\"data row9 col2\" >Mars exploration</td>\n",- " <td id=\"T_070f8_row9_col3\" class=\"data row9 col3\" >❌</td>\n",+ " <th id=\"T_364bd_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",+ " <td id=\"T_364bd_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",+ " <td id=\"T_364bd_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",+ " <td id=\"T_364bd_row9_col2\" class=\"data row9 col2\" >Mars exploration</td>\n",+ " <td id=\"T_364bd_row9_col3\" class=\"data row9 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_070f8_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",- " <td id=\"T_070f8_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",- " <td id=\"T_070f8_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",- " <td id=\"T_070f8_row10_col2\" class=\"data row10 col2\" >Hong Kong</td>\n",- " <td id=\"T_070f8_row10_col3\" class=\"data row10 col3\" >❌</td>\n",+ " <th id=\"T_364bd_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",+ " <td id=\"T_364bd_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",+ " <td id=\"T_364bd_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",+ " <td id=\"T_364bd_row10_col2\" class=\"data row10 col2\" >Hong Kong</td>\n",+ " <td id=\"T_364bd_row10_col3\" class=\"data row10 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_070f8_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",- " <td id=\"T_070f8_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",- " <td id=\"T_070f8_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",- " <td id=\"T_070f8_row11_col2\" class=\"data row11 col2\" >Venus</td>\n",- " <td id=\"T_070f8_row11_col3\" class=\"data row11 col3\" >❌</td>\n",+ " <th id=\"T_364bd_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",+ " <td id=\"T_364bd_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",+ " <td id=\"T_364bd_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",+ " <td id=\"T_364bd_row11_col2\" class=\"data row11 col2\" >Venus</td>\n",+ " <td id=\"T_364bd_row11_col3\" class=\"data row11 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_070f8_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",- " <td id=\"T_070f8_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",- " <td id=\"T_070f8_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",- " <td id=\"T_070f8_row12_col2\" class=\"data row12 col2\" >1457</td>\n",- " <td id=\"T_070f8_row12_col3\" class=\"data row12 col3\" >❌</td>\n",+ " <th id=\"T_364bd_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",+ " <td id=\"T_364bd_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",+ " <td id=\"T_364bd_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",+ " <td id=\"T_364bd_row12_col2\" class=\"data row12 col2\" >1457</td>\n",+ " <td id=\"T_364bd_row12_col3\" class=\"data row12 col3\" >❌</td>\n"," </tr>\n"," </tbody>\n","</table>\n"],"text/plain": [- "<pandas.io.formats.style.Styler at 0x7fa029297fd0>"+ "<pandas.io.formats.style.Styler at 0x7f2d505339a0>"]},"metadata": {},@@ -433,7 +593,7 @@},{"cell_type": "code",- "execution_count": 9,+ "execution_count": 10,"metadata": {},"outputs": [],"source": [@@ -459,7 +619,7 @@},{"cell_type": "code",- "execution_count": 10,+ "execution_count": 11,"metadata": {},"outputs": [],"source": [@@ -483,7 +643,7 @@},{"cell_type": "code",- "execution_count": 11,+ "execution_count": 12,"metadata": {},"outputs": [{@@ -546,7 +706,7 @@"('E. L. Doctorow', None)"]},- "execution_count": 11,+ "execution_count": 12,"metadata": {},"output_type": "execute_result"}@@ -565,14 +725,14 @@},{"cell_type": "code",- "execution_count": 12,+ "execution_count": 13,"metadata": {},"outputs": [{"name": "stderr","output_type": "stream","text": [- "100%|██████████| 13/13 [00:00<00:00, 244.05it/s]"+ "100%|██████████| 13/13 [00:00<00:00, 208.59it/s]"]},{@@ -593,120 +753,120 @@"data": {"text/html": ["<style type=\"text/css\">\n",- "#T_51a84_ th {\n",+ "#T_5ecf5 th {\n"," text-align: left;\n","}\n",- "#T_51a84_ td {\n",+ "#T_5ecf5 td {\n"," text-align: left;\n","}\n","</style>\n",- "<table id=\"T_51a84_\">\n",+ "<table id=\"T_5ecf5\">\n"," <thead>\n"," <tr>\n"," <th class=\"blank level0\" > </th>\n",- " <th class=\"col_heading level0 col0\" >question</th>\n",- " <th class=\"col_heading level0 col1\" >answer</th>\n",- " <th class=\"col_heading level0 col2\" >prediction</th>\n",- " <th class=\"col_heading level0 col3\" >correct</th>\n",+ " <th id=\"T_5ecf5_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",+ " <th id=\"T_5ecf5_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",+ " <th id=\"T_5ecf5_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",+ " <th id=\"T_5ecf5_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n"," </tr>\n"," </thead>\n"," <tbody>\n"," <tr>\n",- " <th id=\"T_51a84_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",- " <td id=\"T_51a84_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",- " <td id=\"T_51a84_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",- " <td id=\"T_51a84_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",- " <td id=\"T_51a84_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",+ " <th id=\"T_5ecf5_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",+ " <td id=\"T_5ecf5_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",+ " <td id=\"T_5ecf5_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",+ " <td id=\"T_5ecf5_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",+ " <td id=\"T_5ecf5_row0_col3\" class=\"data row0 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_51a84_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",- " <td id=\"T_51a84_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",- " <td id=\"T_51a84_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",- " <td id=\"T_51a84_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",- " <td id=\"T_51a84_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",+ " <th id=\"T_5ecf5_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",+ " <td id=\"T_5ecf5_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",+ " <td id=\"T_5ecf5_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",+ " <td id=\"T_5ecf5_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",+ " <td id=\"T_5ecf5_row1_col3\" class=\"data row1 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_51a84_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",- " <td id=\"T_51a84_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",- " <td id=\"T_51a84_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",- " <td id=\"T_51a84_row2_col2\" class=\"data row2 col2\" >Daytona Beach, Florida</td>\n",- " <td id=\"T_51a84_row2_col3\" class=\"data row2 col3\" >❌</td>\n",+ " <th id=\"T_5ecf5_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",+ " <td id=\"T_5ecf5_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",+ " <td id=\"T_5ecf5_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",+ " <td id=\"T_5ecf5_row2_col2\" class=\"data row2 col2\" >Daytona Beach, Florida</td>\n",+ " <td id=\"T_5ecf5_row2_col3\" class=\"data row2 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_51a84_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",- " <td id=\"T_51a84_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",- " <td id=\"T_51a84_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",- " <td id=\"T_51a84_row3_col2\" class=\"data row3 col2\" >1856</td>\n",- " <td id=\"T_51a84_row3_col3\" class=\"data row3 col3\" >❌</td>\n",+ " <th id=\"T_5ecf5_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",+ " <td id=\"T_5ecf5_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",+ " <td id=\"T_5ecf5_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",+ " <td id=\"T_5ecf5_row3_col2\" class=\"data row3 col2\" >1856</td>\n",+ " <td id=\"T_5ecf5_row3_col3\" class=\"data row3 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_51a84_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",- " <td id=\"T_51a84_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",- " <td id=\"T_51a84_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",- " <td id=\"T_51a84_row4_col2\" class=\"data row4 col2\" >Studs Terkel</td>\n",- " <td id=\"T_51a84_row4_col3\" class=\"data row4 col3\" >❌</td>\n",+ " <th id=\"T_5ecf5_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",+ " <td id=\"T_5ecf5_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",+ " <td id=\"T_5ecf5_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",+ " <td id=\"T_5ecf5_row4_col2\" class=\"data row4 col2\" >Studs Terkel</td>\n",+ " <td id=\"T_5ecf5_row4_col3\" class=\"data row4 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_51a84_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",- " <td id=\"T_51a84_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",- " <td id=\"T_51a84_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",- " <td id=\"T_51a84_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",- " <td id=\"T_51a84_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",+ " <th id=\"T_5ecf5_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",+ " <td id=\"T_5ecf5_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",+ " <td id=\"T_5ecf5_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",+ " <td id=\"T_5ecf5_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",+ " <td id=\"T_5ecf5_row5_col3\" class=\"data row5 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_51a84_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",- " <td id=\"T_51a84_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",- " <td id=\"T_51a84_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",- " <td id=\"T_51a84_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",- " <td id=\"T_51a84_row6_col3\" class=\"data row6 col3\" >❌</td>\n",+ " <th id=\"T_5ecf5_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",+ " <td id=\"T_5ecf5_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",+ " <td id=\"T_5ecf5_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",+ " <td id=\"T_5ecf5_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",+ " <td id=\"T_5ecf5_row6_col3\" class=\"data row6 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_51a84_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",- " <td id=\"T_51a84_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",- " <td id=\"T_51a84_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",- " <td id=\"T_51a84_row7_col2\" class=\"data row7 col2\" >5 August 1862</td>\n",- " <td id=\"T_51a84_row7_col3\" class=\"data row7 col3\" >❌</td>\n",+ " <th id=\"T_5ecf5_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",+ " <td id=\"T_5ecf5_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",+ " <td id=\"T_5ecf5_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",+ " <td id=\"T_5ecf5_row7_col2\" class=\"data row7 col2\" >5 August 1862</td>\n",+ " <td id=\"T_5ecf5_row7_col3\" class=\"data row7 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_51a84_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",- " <td id=\"T_51a84_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",- " <td id=\"T_51a84_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",- " <td id=\"T_51a84_row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",- " <td id=\"T_51a84_row8_col3\" class=\"data row8 col3\" >❌</td>\n",+ " <th id=\"T_5ecf5_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",+ " <td id=\"T_5ecf5_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",+ " <td id=\"T_5ecf5_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",+ " <td id=\"T_5ecf5_row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",+ " <td id=\"T_5ecf5_row8_col3\" class=\"data row8 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_51a84_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",- " <td id=\"T_51a84_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",- " <td id=\"T_51a84_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",- " <td id=\"T_51a84_row9_col2\" class=\"data row9 col2\" >Jet Propulsion</td>\n",- " <td id=\"T_51a84_row9_col3\" class=\"data row9 col3\" >❌</td>\n",+ " <th id=\"T_5ecf5_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",+ " <td id=\"T_5ecf5_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",+ " <td id=\"T_5ecf5_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",+ " <td id=\"T_5ecf5_row9_col2\" class=\"data row9 col2\" >Jet Propulsion</td>\n",+ " <td id=\"T_5ecf5_row9_col3\" class=\"data row9 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_51a84_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",- " <td id=\"T_51a84_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",- " <td id=\"T_51a84_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",- " <td id=\"T_51a84_row10_col2\" class=\"data row10 col2\" >Los Angeles, California</td>\n",- " <td id=\"T_51a84_row10_col3\" class=\"data row10 col3\" >❌</td>\n",+ " <th id=\"T_5ecf5_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",+ " <td id=\"T_5ecf5_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",+ " <td id=\"T_5ecf5_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",+ " <td id=\"T_5ecf5_row10_col2\" class=\"data row10 col2\" >Los Angeles, California</td>\n",+ " <td id=\"T_5ecf5_row10_col3\" class=\"data row10 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_51a84_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",- " <td id=\"T_51a84_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",- " <td id=\"T_51a84_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",- " <td id=\"T_51a84_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",- " <td id=\"T_51a84_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",+ " <th id=\"T_5ecf5_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",+ " <td id=\"T_5ecf5_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",+ " <td id=\"T_5ecf5_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",+ " <td id=\"T_5ecf5_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",+ " <td id=\"T_5ecf5_row11_col3\" class=\"data row11 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_51a84_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",- " <td id=\"T_51a84_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",- " <td id=\"T_51a84_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",- " <td id=\"T_51a84_row12_col2\" class=\"data row12 col2\" >1442</td>\n",- " <td id=\"T_51a84_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",+ " <th id=\"T_5ecf5_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",+ " <td id=\"T_5ecf5_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",+ " <td id=\"T_5ecf5_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",+ " <td id=\"T_5ecf5_row12_col2\" class=\"data row12 col2\" >1442</td>\n",+ " <td id=\"T_5ecf5_row12_col3\" class=\"data row12 col3\" >✔️</td>\n"," </tr>\n"," </tbody>\n","</table>\n"],"text/plain": [- "<pandas.io.formats.style.Styler at 0x7fa01eef6d90>"+ "<pandas.io.formats.style.Styler at 0x7f2d5051aee0>"]},"metadata": {},@@ -737,7 +897,7 @@},{"cell_type": "code",- "execution_count": 13,+ "execution_count": 14,"metadata": {},"outputs": [],"source": [@@ -766,7 +926,7 @@},{"cell_type": "code",- "execution_count": 14,+ "execution_count": 15,"metadata": {},"outputs": [],"source": [@@ -798,7 +958,7 @@},{"cell_type": "code",- "execution_count": 15,+ "execution_count": 16,"metadata": {},"outputs": [{@@ -869,7 +1029,7 @@"('Ocala, Florida', None)"]},- "execution_count": 15,+ "execution_count": 16,"metadata": {},"output_type": "execute_result"}@@ -888,14 +1048,14 @@},{"cell_type": "code",- "execution_count": 16,+ "execution_count": 17,"metadata": {},"outputs": [{"name": "stderr","output_type": "stream","text": [- "100%|██████████| 13/13 [00:00<00:00, 18.90it/s]"+ "100%|██████████| 13/13 [00:00<00:00, 16.43it/s]"]},{@@ -916,120 +1076,120 @@"data": {"text/html": ["<style type=\"text/css\">\n",- "#T_2cf39_ th {\n",+ "#T_3695c th {\n"," text-align: left;\n","}\n",- "#T_2cf39_ td {\n",+ "#T_3695c td {\n"," text-align: left;\n","}\n","</style>\n",- "<table id=\"T_2cf39_\">\n",+ "<table id=\"T_3695c\">\n"," <thead>\n"," <tr>\n"," <th class=\"blank level0\" > </th>\n",- " <th class=\"col_heading level0 col0\" >question</th>\n",- " <th class=\"col_heading level0 col1\" >answer</th>\n",- " <th class=\"col_heading level0 col2\" >prediction</th>\n",- " <th class=\"col_heading level0 col3\" >correct</th>\n",+ " <th id=\"T_3695c_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",+ " <th id=\"T_3695c_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",+ " <th id=\"T_3695c_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",+ " <th id=\"T_3695c_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n"," </tr>\n"," </thead>\n"," <tbody>\n"," <tr>\n",- " <th id=\"T_2cf39_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",- " <td id=\"T_2cf39_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",- " <td id=\"T_2cf39_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",- " <td id=\"T_2cf39_row0_col2\" class=\"data row0 col2\" >Julia Peterkin</td>\n",- " <td id=\"T_2cf39_row0_col3\" class=\"data row0 col3\" >❌</td>\n",+ " <th id=\"T_3695c_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",+ " <td id=\"T_3695c_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",+ " <td id=\"T_3695c_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",+ " <td id=\"T_3695c_row0_col2\" class=\"data row0 col2\" >Julia Peterkin</td>\n",+ " <td id=\"T_3695c_row0_col3\" class=\"data row0 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_2cf39_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",- " <td id=\"T_2cf39_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",- " <td id=\"T_2cf39_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",- " <td id=\"T_2cf39_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",- " <td id=\"T_2cf39_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",+ " <th id=\"T_3695c_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",+ " <td id=\"T_3695c_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",+ " <td id=\"T_3695c_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",+ " <td id=\"T_3695c_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",+ " <td id=\"T_3695c_row1_col3\" class=\"data row1 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_2cf39_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",- " <td id=\"T_2cf39_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",- " <td id=\"T_2cf39_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",- " <td id=\"T_2cf39_row2_col2\" class=\"data row2 col2\" >Ocala, Florida</td>\n",- " <td id=\"T_2cf39_row2_col3\" class=\"data row2 col3\" >❌</td>\n",+ " <th id=\"T_3695c_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",+ " <td id=\"T_3695c_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",+ " <td id=\"T_3695c_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",+ " <td id=\"T_3695c_row2_col2\" class=\"data row2 col2\" >Ocala, Florida</td>\n",+ " <td id=\"T_3695c_row2_col3\" class=\"data row2 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_2cf39_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",- " <td id=\"T_2cf39_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",- " <td id=\"T_2cf39_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",- " <td id=\"T_2cf39_row3_col2\" class=\"data row3 col2\" >1824</td>\n",- " <td id=\"T_2cf39_row3_col3\" class=\"data row3 col3\" >❌</td>\n",+ " <th id=\"T_3695c_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",+ " <td id=\"T_3695c_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",+ " <td id=\"T_3695c_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",+ " <td id=\"T_3695c_row3_col2\" class=\"data row3 col2\" >1824</td>\n",+ " <td id=\"T_3695c_row3_col3\" class=\"data row3 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_2cf39_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",- " <td id=\"T_2cf39_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",- " <td id=\"T_2cf39_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",- " <td id=\"T_2cf39_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",- " <td id=\"T_2cf39_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",+ " <th id=\"T_3695c_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",+ " <td id=\"T_3695c_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",+ " <td id=\"T_3695c_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",+ " <td id=\"T_3695c_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",+ " <td id=\"T_3695c_row4_col3\" class=\"data row4 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_2cf39_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",- " <td id=\"T_2cf39_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",- " <td id=\"T_2cf39_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",- " <td id=\"T_2cf39_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",- " <td id=\"T_2cf39_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",+ " <th id=\"T_3695c_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",+ " <td id=\"T_3695c_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",+ " <td id=\"T_3695c_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",+ " <td id=\"T_3695c_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",+ " <td id=\"T_3695c_row5_col3\" class=\"data row5 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_2cf39_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",- " <td id=\"T_2cf39_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",- " <td id=\"T_2cf39_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",- " <td id=\"T_2cf39_row6_col2\" class=\"data row6 col2\" >Cheshire, UK</td>\n",- " <td id=\"T_2cf39_row6_col3\" class=\"data row6 col3\" >✔️</td>\n",+ " <th id=\"T_3695c_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",+ " <td id=\"T_3695c_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",+ " <td id=\"T_3695c_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",+ " <td id=\"T_3695c_row6_col2\" class=\"data row6 col2\" >Cheshire, UK</td>\n",+ " <td id=\"T_3695c_row6_col3\" class=\"data row6 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_2cf39_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",- " <td id=\"T_2cf39_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",- " <td id=\"T_2cf39_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",- " <td id=\"T_2cf39_row7_col2\" class=\"data row7 col2\" >September 27, 1974</td>\n",- " <td id=\"T_2cf39_row7_col3\" class=\"data row7 col3\" >❌</td>\n",+ " <th id=\"T_3695c_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",+ " <td id=\"T_3695c_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",+ " <td id=\"T_3695c_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",+ " <td id=\"T_3695c_row7_col2\" class=\"data row7 col2\" >September 27, 1974</td>\n",+ " <td id=\"T_3695c_row7_col3\" class=\"data row7 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_2cf39_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",- " <td id=\"T_2cf39_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",- " <td id=\"T_2cf39_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",- " <td id=\"T_2cf39_row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",- " <td id=\"T_2cf39_row8_col3\" class=\"data row8 col3\" >❌</td>\n",+ " <th id=\"T_3695c_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",+ " <td id=\"T_3695c_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",+ " <td id=\"T_3695c_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",+ " <td id=\"T_3695c_row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",+ " <td id=\"T_3695c_row8_col3\" class=\"data row8 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_2cf39_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",- " <td id=\"T_2cf39_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",- " <td id=\"T_2cf39_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",- " <td id=\"T_2cf39_row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",- " <td id=\"T_2cf39_row9_col3\" class=\"data row9 col3\" >❌</td>\n",+ " <th id=\"T_3695c_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",+ " <td id=\"T_3695c_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",+ " <td id=\"T_3695c_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",+ " <td id=\"T_3695c_row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",+ " <td id=\"T_3695c_row9_col3\" class=\"data row9 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_2cf39_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",- " <td id=\"T_2cf39_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",- " <td id=\"T_2cf39_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",- " <td id=\"T_2cf39_row10_col2\" class=\"data row10 col2\" >Italy</td>\n",- " <td id=\"T_2cf39_row10_col3\" class=\"data row10 col3\" >❌</td>\n",+ " <th id=\"T_3695c_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",+ " <td id=\"T_3695c_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",+ " <td id=\"T_3695c_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",+ " <td id=\"T_3695c_row10_col2\" class=\"data row10 col2\" >Italy</td>\n",+ " <td id=\"T_3695c_row10_col3\" class=\"data row10 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_2cf39_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",- " <td id=\"T_2cf39_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",- " <td id=\"T_2cf39_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",- " <td id=\"T_2cf39_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",- " <td id=\"T_2cf39_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",+ " <th id=\"T_3695c_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",+ " <td id=\"T_3695c_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",+ " <td id=\"T_3695c_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",+ " <td id=\"T_3695c_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",+ " <td id=\"T_3695c_row11_col3\" class=\"data row11 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_2cf39_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",- " <td id=\"T_2cf39_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",- " <td id=\"T_2cf39_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",- " <td id=\"T_2cf39_row12_col2\" class=\"data row12 col2\" >1442</td>\n",- " <td id=\"T_2cf39_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",+ " <th id=\"T_3695c_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",+ " <td id=\"T_3695c_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",+ " <td id=\"T_3695c_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",+ " <td id=\"T_3695c_row12_col2\" class=\"data row12 col2\" >1442</td>\n",+ " <td id=\"T_3695c_row12_col3\" class=\"data row12 col3\" >✔️</td>\n"," </tr>\n"," </tbody>\n","</table>\n"],"text/plain": [- "<pandas.io.formats.style.Styler at 0x7fa01e93e3d0>"+ "<pandas.io.formats.style.Styler at 0x7f2c97dcf2b0>"]},"metadata": {},@@ -1064,7 +1224,7 @@},{"cell_type": "code",- "execution_count": 17,+ "execution_count": 18,"metadata": {},"outputs": [],"source": [@@ -1094,7 +1254,7 @@},{"cell_type": "code",- "execution_count": 18,+ "execution_count": 19,"metadata": {},"outputs": [],"source": [@@ -1123,7 +1283,7 @@},{"cell_type": "code",- "execution_count": 19,+ "execution_count": 20,"metadata": {},"outputs": [],"source": [@@ -1167,7 +1327,7 @@},{"cell_type": "code",- "execution_count": 20,+ "execution_count": 21,"metadata": {},"outputs": [{@@ -1292,7 +1452,7 @@"('Gainesville, Florida', None)"]},- "execution_count": 20,+ "execution_count": 21,"metadata": {},"output_type": "execute_result"}@@ -1311,14 +1471,14 @@},{"cell_type": "code",- "execution_count": 21,+ "execution_count": 22,"metadata": {},"outputs": [{"name": "stderr","output_type": "stream","text": [- "100%|██████████| 13/13 [00:00<00:00, 89.68it/s]"+ "100%|██████████| 13/13 [00:00<00:00, 83.91it/s]"]},{@@ -1339,120 +1499,120 @@"data": {"text/html": ["<style type=\"text/css\">\n",- "#T_f09c3_ th {\n",+ "#T_83eb7 th {\n"," text-align: left;\n","}\n",- "#T_f09c3_ td {\n",+ "#T_83eb7 td {\n"," text-align: left;\n","}\n","</style>\n",- "<table id=\"T_f09c3_\">\n",+ "<table id=\"T_83eb7\">\n"," <thead>\n"," <tr>\n"," <th class=\"blank level0\" > </th>\n",- " <th class=\"col_heading level0 col0\" >question</th>\n",- " <th class=\"col_heading level0 col1\" >answer</th>\n",- " <th class=\"col_heading level0 col2\" >prediction</th>\n",- " <th class=\"col_heading level0 col3\" >correct</th>\n",+ " <th id=\"T_83eb7_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",+ " <th id=\"T_83eb7_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",+ " <th id=\"T_83eb7_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",+ " <th id=\"T_83eb7_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n"," </tr>\n"," </thead>\n"," <tbody>\n"," <tr>\n",- " <th id=\"T_f09c3_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",- " <td id=\"T_f09c3_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",- " <td id=\"T_f09c3_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",- " <td id=\"T_f09c3_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",- " <td id=\"T_f09c3_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",+ " <th id=\"T_83eb7_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",+ " <td id=\"T_83eb7_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",+ " <td id=\"T_83eb7_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",+ " <td id=\"T_83eb7_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",+ " <td id=\"T_83eb7_row0_col3\" class=\"data row0 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_f09c3_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",- " <td id=\"T_f09c3_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",- " <td id=\"T_f09c3_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",- " <td id=\"T_f09c3_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",- " <td id=\"T_f09c3_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",+ " <th id=\"T_83eb7_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",+ " <td id=\"T_83eb7_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",+ " <td id=\"T_83eb7_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",+ " <td id=\"T_83eb7_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",+ " <td id=\"T_83eb7_row1_col3\" class=\"data row1 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_f09c3_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",- " <td id=\"T_f09c3_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",- " <td id=\"T_f09c3_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",- " <td id=\"T_f09c3_row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",- " <td id=\"T_f09c3_row2_col3\" class=\"data row2 col3\" >✔️</td>\n",+ " <th id=\"T_83eb7_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",+ " <td id=\"T_83eb7_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",+ " <td id=\"T_83eb7_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",+ " <td id=\"T_83eb7_row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",+ " <td id=\"T_83eb7_row2_col3\" class=\"data row2 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_f09c3_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",- " <td id=\"T_f09c3_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",- " <td id=\"T_f09c3_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",- " <td id=\"T_f09c3_row3_col2\" class=\"data row3 col2\" >1947</td>\n",- " <td id=\"T_f09c3_row3_col3\" class=\"data row3 col3\" >❌</td>\n",+ " <th id=\"T_83eb7_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",+ " <td id=\"T_83eb7_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",+ " <td id=\"T_83eb7_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",+ " <td id=\"T_83eb7_row3_col2\" class=\"data row3 col2\" >1947</td>\n",+ " <td id=\"T_83eb7_row3_col3\" class=\"data row3 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_f09c3_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",- " <td id=\"T_f09c3_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",- " <td id=\"T_f09c3_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",- " <td id=\"T_f09c3_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",- " <td id=\"T_f09c3_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",+ " <th id=\"T_83eb7_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",+ " <td id=\"T_83eb7_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",+ " <td id=\"T_83eb7_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",+ " <td id=\"T_83eb7_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",+ " <td id=\"T_83eb7_row4_col3\" class=\"data row4 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_f09c3_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",- " <td id=\"T_f09c3_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",- " <td id=\"T_f09c3_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",- " <td id=\"T_f09c3_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",- " <td id=\"T_f09c3_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",+ " <th id=\"T_83eb7_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",+ " <td id=\"T_83eb7_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",+ " <td id=\"T_83eb7_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",+ " <td id=\"T_83eb7_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",+ " <td id=\"T_83eb7_row5_col3\" class=\"data row5 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_f09c3_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",- " <td id=\"T_f09c3_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",- " <td id=\"T_f09c3_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",- " <td id=\"T_f09c3_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",- " <td id=\"T_f09c3_row6_col3\" class=\"data row6 col3\" >❌</td>\n",+ " <th id=\"T_83eb7_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",+ " <td id=\"T_83eb7_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",+ " <td id=\"T_83eb7_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",+ " <td id=\"T_83eb7_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",+ " <td id=\"T_83eb7_row6_col3\" class=\"data row6 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_f09c3_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",- " <td id=\"T_f09c3_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",- " <td id=\"T_f09c3_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",- " <td id=\"T_f09c3_row7_col2\" class=\"data row7 col2\" >1965</td>\n",- " <td id=\"T_f09c3_row7_col3\" class=\"data row7 col3\" >❌</td>\n",+ " <th id=\"T_83eb7_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",+ " <td id=\"T_83eb7_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",+ " <td id=\"T_83eb7_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",+ " <td id=\"T_83eb7_row7_col2\" class=\"data row7 col2\" >1965</td>\n",+ " <td id=\"T_83eb7_row7_col3\" class=\"data row7 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_f09c3_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",- " <td id=\"T_f09c3_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",- " <td id=\"T_f09c3_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",- " <td id=\"T_f09c3_row8_col2\" class=\"data row8 col2\" >River Foyle</td>\n",- " <td id=\"T_f09c3_row8_col3\" class=\"data row8 col3\" >❌</td>\n",+ " <th id=\"T_83eb7_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",+ " <td id=\"T_83eb7_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",+ " <td id=\"T_83eb7_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",+ " <td id=\"T_83eb7_row8_col2\" class=\"data row8 col2\" >River Foyle</td>\n",+ " <td id=\"T_83eb7_row8_col3\" class=\"data row8 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_f09c3_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",- " <td id=\"T_f09c3_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",- " <td id=\"T_f09c3_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",- " <td id=\"T_f09c3_row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",- " <td id=\"T_f09c3_row9_col3\" class=\"data row9 col3\" >❌</td>\n",+ " <th id=\"T_83eb7_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",+ " <td id=\"T_83eb7_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",+ " <td id=\"T_83eb7_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",+ " <td id=\"T_83eb7_row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",+ " <td id=\"T_83eb7_row9_col3\" class=\"data row9 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_f09c3_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",- " <td id=\"T_f09c3_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",- " <td id=\"T_f09c3_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",- " <td id=\"T_f09c3_row10_col2\" class=\"data row10 col2\" >Guildford, Surrey</td>\n",- " <td id=\"T_f09c3_row10_col3\" class=\"data row10 col3\" >✔️</td>\n",+ " <th id=\"T_83eb7_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",+ " <td id=\"T_83eb7_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",+ " <td id=\"T_83eb7_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",+ " <td id=\"T_83eb7_row10_col2\" class=\"data row10 col2\" >Guildford, Surrey</td>\n",+ " <td id=\"T_83eb7_row10_col3\" class=\"data row10 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_f09c3_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",- " <td id=\"T_f09c3_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",- " <td id=\"T_f09c3_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",- " <td id=\"T_f09c3_row11_col2\" class=\"data row11 col2\" >Carrington</td>\n",- " <td id=\"T_f09c3_row11_col3\" class=\"data row11 col3\" >❌</td>\n",+ " <th id=\"T_83eb7_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",+ " <td id=\"T_83eb7_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",+ " <td id=\"T_83eb7_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",+ " <td id=\"T_83eb7_row11_col2\" class=\"data row11 col2\" >Carrington</td>\n",+ " <td id=\"T_83eb7_row11_col3\" class=\"data row11 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_f09c3_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",- " <td id=\"T_f09c3_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",- " <td id=\"T_f09c3_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",- " <td id=\"T_f09c3_row12_col2\" class=\"data row12 col2\" >1442</td>\n",- " <td id=\"T_f09c3_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",+ " <th id=\"T_83eb7_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",+ " <td id=\"T_83eb7_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",+ " <td id=\"T_83eb7_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",+ " <td id=\"T_83eb7_row12_col2\" class=\"data row12 col2\" >1442</td>\n",+ " <td id=\"T_83eb7_row12_col3\" class=\"data row12 col3\" >✔️</td>\n"," </tr>\n"," </tbody>\n","</table>\n"],"text/plain": [- "<pandas.io.formats.style.Styler at 0x7fa01d1be810>"+ "<pandas.io.formats.style.Styler at 0x7f2d50533ee0>"]},"metadata": {},@@ -1489,7 +1649,7 @@},{"cell_type": "code",- "execution_count": 22,+ "execution_count": 23,"metadata": {},"outputs": [],"source": [@@ -1526,7 +1686,7 @@},{"cell_type": "code",- "execution_count": 23,+ "execution_count": 24,"metadata": {},"outputs": [],"source": [@@ -1559,7 +1719,7 @@},{"cell_type": "code",- "execution_count": 24,+ "execution_count": 25,"metadata": {},"outputs": [{@@ -1777,7 +1937,7 @@"('1828', None)"]},- "execution_count": 24,+ "execution_count": 25,"metadata": {},"output_type": "execute_result"}@@ -1788,14 +1948,14 @@},{"cell_type": "code",- "execution_count": 25,+ "execution_count": 26,"metadata": {},"outputs": [{"name": "stderr","output_type": "stream","text": [- "100%|██████████| 13/13 [00:01<00:00, 8.38it/s]"+ "100%|██████████| 13/13 [00:01<00:00, 8.50it/s]"]},{@@ -1816,120 +1976,120 @@"data": {"text/html": ["<style type=\"text/css\">\n",- "#T_b8b4e_ th {\n",+ "#T_b6174 th {\n"," text-align: left;\n","}\n",- "#T_b8b4e_ td {\n",+ "#T_b6174 td {\n"," text-align: left;\n","}\n","</style>\n",- "<table id=\"T_b8b4e_\">\n",+ "<table id=\"T_b6174\">\n"," <thead>\n"," <tr>\n"," <th class=\"blank level0\" > </th>\n",- " <th class=\"col_heading level0 col0\" >question</th>\n",- " <th class=\"col_heading level0 col1\" >answer</th>\n",- " <th class=\"col_heading level0 col2\" >prediction</th>\n",- " <th class=\"col_heading level0 col3\" >correct</th>\n",+ " <th id=\"T_b6174_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",+ " <th id=\"T_b6174_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",+ " <th id=\"T_b6174_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",+ " <th id=\"T_b6174_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n"," </tr>\n"," </thead>\n"," <tbody>\n"," <tr>\n",- " <th id=\"T_b8b4e_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",- " <td id=\"T_b8b4e_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",- " <td id=\"T_b8b4e_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",- " <td id=\"T_b8b4e_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",- " <td id=\"T_b8b4e_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",+ " <th id=\"T_b6174_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",+ " <td id=\"T_b6174_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",+ " <td id=\"T_b6174_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",+ " <td id=\"T_b6174_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",+ " <td id=\"T_b6174_row0_col3\" class=\"data row0 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_b8b4e_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",- " <td id=\"T_b8b4e_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",- " <td id=\"T_b8b4e_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",- " <td id=\"T_b8b4e_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",- " <td id=\"T_b8b4e_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",+ " <th id=\"T_b6174_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",+ " <td id=\"T_b6174_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",+ " <td id=\"T_b6174_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",+ " <td id=\"T_b6174_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",+ " <td id=\"T_b6174_row1_col3\" class=\"data row1 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_b8b4e_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",- " <td id=\"T_b8b4e_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",- " <td id=\"T_b8b4e_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",- " <td id=\"T_b8b4e_row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",- " <td id=\"T_b8b4e_row2_col3\" class=\"data row2 col3\" >✔️</td>\n",+ " <th id=\"T_b6174_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",+ " <td id=\"T_b6174_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",+ " <td id=\"T_b6174_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",+ " <td id=\"T_b6174_row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",+ " <td id=\"T_b6174_row2_col3\" class=\"data row2 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_b8b4e_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",- " <td id=\"T_b8b4e_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",- " <td id=\"T_b8b4e_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",- " <td id=\"T_b8b4e_row3_col2\" class=\"data row3 col2\" >1828</td>\n",- " <td id=\"T_b8b4e_row3_col3\" class=\"data row3 col3\" >✔️</td>\n",+ " <th id=\"T_b6174_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",+ " <td id=\"T_b6174_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",+ " <td id=\"T_b6174_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",+ " <td id=\"T_b6174_row3_col2\" class=\"data row3 col2\" >1828</td>\n",+ " <td id=\"T_b6174_row3_col3\" class=\"data row3 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_b8b4e_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",- " <td id=\"T_b8b4e_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",- " <td id=\"T_b8b4e_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",- " <td id=\"T_b8b4e_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",- " <td id=\"T_b8b4e_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",+ " <th id=\"T_b6174_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",+ " <td id=\"T_b6174_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",+ " <td id=\"T_b6174_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",+ " <td id=\"T_b6174_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",+ " <td id=\"T_b6174_row4_col3\" class=\"data row4 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_b8b4e_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",- " <td id=\"T_b8b4e_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",- " <td id=\"T_b8b4e_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",- " <td id=\"T_b8b4e_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",- " <td id=\"T_b8b4e_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",+ " <th id=\"T_b6174_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",+ " <td id=\"T_b6174_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",+ " <td id=\"T_b6174_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",+ " <td id=\"T_b6174_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",+ " <td id=\"T_b6174_row5_col3\" class=\"data row5 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_b8b4e_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",- " <td id=\"T_b8b4e_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",- " <td id=\"T_b8b4e_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",- " <td id=\"T_b8b4e_row6_col2\" class=\"data row6 col2\" >Cheshire</td>\n",- " <td id=\"T_b8b4e_row6_col3\" class=\"data row6 col3\" >✔️</td>\n",+ " <th id=\"T_b6174_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",+ " <td id=\"T_b6174_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",+ " <td id=\"T_b6174_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",+ " <td id=\"T_b6174_row6_col2\" class=\"data row6 col2\" >Cheshire</td>\n",+ " <td id=\"T_b6174_row6_col3\" class=\"data row6 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_b8b4e_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",- " <td id=\"T_b8b4e_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",- " <td id=\"T_b8b4e_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",- " <td id=\"T_b8b4e_row7_col2\" class=\"data row7 col2\" >1958</td>\n",- " <td id=\"T_b8b4e_row7_col3\" class=\"data row7 col3\" >❌</td>\n",+ " <th id=\"T_b6174_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",+ " <td id=\"T_b6174_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",+ " <td id=\"T_b6174_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",+ " <td id=\"T_b6174_row7_col2\" class=\"data row7 col2\" >1958</td>\n",+ " <td id=\"T_b6174_row7_col3\" class=\"data row7 col3\" >❌</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_b8b4e_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",- " <td id=\"T_b8b4e_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",- " <td id=\"T_b8b4e_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",- " <td id=\"T_b8b4e_row8_col2\" class=\"data row8 col2\" >Castletown River</td>\n",- " <td id=\"T_b8b4e_row8_col3\" class=\"data row8 col3\" >✔️</td>\n",+ " <th id=\"T_b6174_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",+ " <td id=\"T_b6174_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",+ " <td id=\"T_b6174_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",+ " <td id=\"T_b6174_row8_col2\" class=\"data row8 col2\" >Castletown River</td>\n",+ " <td id=\"T_b6174_row8_col3\" class=\"data row8 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_b8b4e_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",- " <td id=\"T_b8b4e_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",- " <td id=\"T_b8b4e_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",- " <td id=\"T_b8b4e_row9_col2\" class=\"data row9 col2\" >Aeronautics</td>\n",- " <td id=\"T_b8b4e_row9_col3\" class=\"data row9 col3\" >✔️</td>\n",+ " <th id=\"T_b6174_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",+ " <td id=\"T_b6174_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",+ " <td id=\"T_b6174_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",+ " <td id=\"T_b6174_row9_col2\" class=\"data row9 col2\" >Aeronautics</td>\n",+ " <td id=\"T_b6174_row9_col3\" class=\"data row9 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_b8b4e_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",- " <td id=\"T_b8b4e_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",- " <td id=\"T_b8b4e_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",- " <td id=\"T_b8b4e_row10_col2\" class=\"data row10 col2\" >Surrey</td>\n",- " <td id=\"T_b8b4e_row10_col3\" class=\"data row10 col3\" >✔️</td>\n",+ " <th id=\"T_b6174_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",+ " <td id=\"T_b6174_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",+ " <td id=\"T_b6174_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",+ " <td id=\"T_b6174_row10_col2\" class=\"data row10 col2\" >Surrey</td>\n",+ " <td id=\"T_b6174_row10_col3\" class=\"data row10 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_b8b4e_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",- " <td id=\"T_b8b4e_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",- " <td id=\"T_b8b4e_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",- " <td id=\"T_b8b4e_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",- " <td id=\"T_b8b4e_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",+ " <th id=\"T_b6174_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",+ " <td id=\"T_b6174_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",+ " <td id=\"T_b6174_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",+ " <td id=\"T_b6174_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",+ " <td id=\"T_b6174_row11_col3\" class=\"data row11 col3\" >✔️</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_b8b4e_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",- " <td id=\"T_b8b4e_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",- " <td id=\"T_b8b4e_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",- " <td id=\"T_b8b4e_row12_col2\" class=\"data row12 col2\" >1442</td>\n",- " <td id=\"T_b8b4e_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",+ " <th id=\"T_b6174_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",+ " <td id=\"T_b6174_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",+ " <td id=\"T_b6174_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",+ " <td id=\"T_b6174_row12_col2\" class=\"data row12 col2\" >1442</td>\n",+ " <td id=\"T_b6174_row12_col3\" class=\"data row12 col3\" >✔️</td>\n"," </tr>\n"," </tbody>\n","</table>\n"],"text/plain": [- "<pandas.io.formats.style.Styler at 0x7fa01d1dbb50>"+ "<pandas.io.formats.style.Styler at 0x7f2c9404ee80>"]},"metadata": {},@@ -1945,15 +2105,39 @@"cell_type": "markdown","metadata": {},"source": [+ "### Additional Programs...\n",+ "\n","While Program 5 is undeniably better than, say, Program 1, it's far from perfect.\n","\n",- "Many different ideas can easily be implemented to make it more general or more powerful. Refer to the DSP paper for inspiration!"+ "Many different ideas can easily be implemented to make it more general or more powerful. Refer to the DSP paper for inspiration!\n",+ "\n",+ "One of your first improvements to Programs 4 and 5 would be to allow it to do more than two hops and to automatically stop when additional hops are not required.\n",+ "\n",+ "**Hint:** This is pretty easy to do. You can start with Program 4 and modify `hop_template` so its `SearchQuery` produces the string `N/A` when additional hops are not required.\n",+ "\n",+ "In the `multihop_search_v1` function, you can then break from the loop — i.e., `if completions.query == 'N/A': break` whenever this happens. How will you apply this to Program 5?\n",+ "\n",+ "```\n",+ "SearchQueryWithStopping = dsp.Type(\n",+ " prefix=\"Search Query:\",\n",+ " desc=\"${a simple question for seeking the missing or required information --- say N/A if the context above contains all of the required information}\"\n",+ ")\n",+ "```"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 27,+ "metadata": {},+ "outputs": [],+ "source": [+ "# Program 4 with max_hops=4 and automatic stopping."]}],"metadata": {"kernelspec": {- "display_name": "colbert-api-001",+ "display_name": "py38","language": "python","name": "python3"},@@ -1967,12 +2151,12 @@"name": "python","nbconvert_exporter": "python","pygments_lexer": "ipython3",- "version": "3.7.11"+ "version": "3.8.16"},"orig_nbformat": 4,"vscode": {"interpreter": {- "hash": "08fdf29897c5b8f3cfe29625f379bb8bd67a7135431cc64e36a1afcc33cadfae"+ "hash": "ff8889dd26469456638524e92174013821322a2ab8e175954d0360930982a899"}}},