Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Add first demo program

Rick Battlee8f679968277
diff --git a/demos/Multi-hop QA Program 1 - Direct Query.ipynb b/demos/Multi-hop QA Program 1 - Direct Query.ipynb
new file mode 100644
index 000000000..fd9d058b7
--- /dev/null
+++ b/demos/Multi-hop QA Program 1 - Direct Query.ipynb
@@ -0,0 +1,489 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "551cc8d5-fd87-4d94-ba16-dc2a48698a6f",
+ "metadata": {},
+ "source": [
+ "# Multi-hop QA Program 1: Direct Query\n",
+ "\n",
+ "This notebook is a stand-alone version of Program 1 from the intro notebook."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4e59bc24-3aec-459d-a1a5-e98717431907",
+ "metadata": {},
+ "source": [
+ "### Installation\n",
+ "\n",
+ "If you haven't installed **DSP** already, let's do that.\n",
+ "\n",
+ "Note: If you're running this from a cloned copy of the repo, then you can skip this block."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "4834f1e4-c3b8-498e-9376-4241c3d2324f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
+ " import google.colab \n",
+ " !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",
+ "except: pass\n",
+ "\n",
+ "!pip install -U pip dsp-ml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7d71eb82-0b21-4982-9b5f-f2ac6a8e5611",
+ "metadata": {},
+ "source": [
+ "### Setting Up\n",
+ "\n",
+ "We'll start by setting up the language model (LM) and retrieval model (RM).\n",
+ "\n",
+ "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",
+ "\n",
+ "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",
+ "\n",
+ "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "2b542158-bc91-48e2-b201-638c9e1ea606",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Not loading Cohere because it is not installed.\n"
+ ]
+ }
+ ],
+ "source": [
+ "%load_ext autoreload\n",
+ "%autoreload 2\n",
+ "\n",
+ "try: import google.colab; root_path = 'dsp'\n",
+ "# The root path is ../ if you're running this from the demo folder of the cloned repository\n",
+ "except: root_path = '../'\n",
+ "\n",
+ "import os\n",
+ "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(root_path, 'cache')\n",
+ "\n",
+ "# Add ../ to the path to import dsp if you're running this directly from the cloned copy of the repo (without pip installing dsp)\n",
+ "import sys\n",
+ "sys.path.insert(0, '../')\n",
+ "\n",
+ "import dsp\n",
+ "\n",
+ "openai_key = os.getenv('OPENAI_API_KEY') # or replace with your API key (optional)\n",
+ "colbert_server = 'http://ec2-44-228-128-229.us-west-2.compute.amazonaws.com:8893/api/search'\n",
+ "\n",
+ "lm = dsp.GPT3(model='text-davinci-002', api_key=openai_key)\n",
+ "rm = dsp.ColBERTv2(url=colbert_server)\n",
+ "\n",
+ "dsp.settings.configure(lm=lm, rm=rm)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b3289955-e877-4b3e-8b8c-5e9e5a1d2921",
+ "metadata": {},
+ "source": [
+ "### Task Examples\n",
+ "\n",
+ "Next, let's look at a few examples of the task. Each example consists of a question and one or more gold answers.\n",
+ "\n",
+ "We have six training examples (`train`), which we'll feed into the programs. These will help define the task.\n",
+ "\n",
+ "Notice that our examples only have input (`question`) and output (`answer`) fields. When our advanced programs build sophisticated pipelines, training \"demonstrations\" for other fields will be constructed automatically."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "ad480ffd-8299-4b36-81d9-493a27d1b28d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "train = [('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",
+ " ('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",
+ " ('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",
+ " ('In what year was the star of To Hell and Back born?', ['1925']),\n",
+ " ('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",
+ " ('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),]\n",
+ "\n",
+ "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2870fe3d-ffb8-4214-9279-ac040aad1942",
+ "metadata": {},
+ "source": [
+ "The development examples (`dev`) will be used to assess the behavior of each program we build. Of course, this tiny set is not meant to be a reliable benchmark, but it'll be instructive to use it for illustration."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "2ebf9b6d-4dfd-4023-8ded-64dfc11348ae",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "dev = [('Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?', ['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']),\n",
+ " ('What documentary about the Gilgo Beach Killer debuted on A&E?', ['The Killing Season']),\n",
+ " ('Right Back At It Again contains lyrics co-written by the singer born in what city?', ['Gainesville, Florida', 'Gainesville']),\n",
+ " ('What year was the party of the winner of the 1971 San Francisco mayoral election founded?', ['1828']),\n",
+ " ('Which author is English: John Braine or Studs Terkel?', ['John Braine']),\n",
+ " ('Anthony Dirrell is the brother of which super middleweight title holder?', ['Andre Dirrell']),\n",
+ " ('In which city is the sports nutrition business established by Oliver Cookson based ?', ['Cheshire', 'Cheshire, UK']),\n",
+ " ('Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.', ['February 13, 1980']),\n",
+ " ('Kyle Moran was born in the town on what river?', ['Castletown', 'Castletown River']),\n",
+ " (\"What is the name of one branch of Robert D. Braun's speciality?\", ['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']),\n",
+ " (\"Where was the actress who played the niece in the Priest film born?\", ['Surrey', 'Guildford, Surrey']),\n",
+ " ('Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.', ['Portrait of a Marriage']),\n",
+ " ('What year was the father of the Princes in the Tower born?', ['1442'])]\n",
+ "\n",
+ "dev = [dsp.Example(question=question, answer=answer) for question, answer in dev]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d7df81ab-9c30-43ec-abad-25b5a58f0cf1",
+ "metadata": {},
+ "source": [
+ "### Program Definition\n",
+ "\n",
+ "Direct Query is the simplest program for this task. We'll prompt **GPT-3.5** to answer each question based on its internal parameteric knowledge.\n",
+ "\n",
+ "We'll start by defining the `Template` that defines how we will communicate with the LM.\n",
+ "\n",
+ "Specifically, the question–answer template (`qa_template`) will include a question and a short answer for each example."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "72d65b15-443d-4e3e-b327-b19a396ba5ae",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "Question = dsp.Type(prefix=\"Question:\", desc=\"${the question to be answered}\")\n",
+ "Answer = dsp.Type(prefix=\"Answer:\", desc=\"${a short factoid answer, often between 1 and 5 words}\", format=dsp.format_answers)\n",
+ "\n",
+ "qa_template = dsp.Template(instructions=\"Answer questions with short factoid answers.\", question=Question(), answer=Answer())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4610915b-fad0-4ecf-b994-b9df035e8845",
+ "metadata": {},
+ "source": [
+ "Then, let's define the actual program, `Direct_Query_QA`. It'll accept a string (`question`) and returns another string (its short `answer`)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "926e9edb-2447-4b32-85e9-4105d5b0c39d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def Direct_Query_QA(question: str) -> str:\n",
+ " demos = dsp.sample(train, k=7)\n",
+ " example = dsp.Example(question=question, demos=demos)\n",
+ "\n",
+ " example, completions = dsp.generate(qa_template)(example, stage='qa')\n",
+ " return completions.answer"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e5ea19e3-5a7c-4984-a9b3-eab408244912",
+ "metadata": {},
+ "source": [
+ "Let's invoke the program on a sample question."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "c3c96381-6dab-4214-8faf-1c649de5c5c8",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
+ "E. L. Doctorow\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(dev[0].question)\n",
+ "print(Direct_Query_QA(dev[0].question))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "424c4ffb-4e71-4a91-a179-7cacadcbc2f5",
+ "metadata": {},
+ "source": [
+ "Let's inspect the last call to the LM to learn more about the behavior of the program."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "f8fe21c1-e185-44c3-8742-7fb19385e8a9",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Answer questions with short factoid answers.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Question: ${the question to be answered}\n",
+ "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: Which award did the first book of Gary Zukav receive?\n",
+ "Answer: U.S. National Book Award\n",
+ "\n",
+ "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "Answer: Foxcatcher\n",
+ "\n",
+ "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "Answer: Kevin Greutert\n",
+ "\n",
+ "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
+ "Answer: Butch Vig\n",
+ "\n",
+ "Question: What city was the victim of Joseph Druces working in?\n",
+ "Answer: Boston, Massachusetts\n",
+ "\n",
+ "Question: In what year was the star of To Hell and Back born?\n",
+ "Answer: 1925\n",
+ "\n",
+ "Question: Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
+ "Answer:\u001b[32m E. L. Doctorow\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "lm.inspect_history(n=1)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "id": "886ff6e7-0ca9-4dff-a042-5357915924ac",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 13/13 [00:00<00:00, 121.15it/s]"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Answered 3 / 13 (23.1%) correctly.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "<style type=\"text/css\">\n",
+ "#T_60df5 th {\n",
+ " text-align: left;\n",
+ "}\n",
+ "#T_60df5 td {\n",
+ " text-align: left;\n",
+ "}\n",
+ "</style>\n",
+ "<table id=\"T_60df5\">\n",
+ " <thead>\n",
+ " <tr>\n",
+ " <th class=\"blank level0\" >&nbsp;</th>\n",
+ " <th id=\"T_60df5_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
+ " <th id=\"T_60df5_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
+ " <th id=\"T_60df5_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
+ " <th id=\"T_60df5_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th id=\"T_60df5_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
+ " <td id=\"T_60df5_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
+ " <td id=\"T_60df5_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
+ " <td id=\"T_60df5_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
+ " <td id=\"T_60df5_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_60df5_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
+ " <td id=\"T_60df5_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
+ " <td id=\"T_60df5_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
+ " <td id=\"T_60df5_row1_col2\" class=\"data row1 col2\" >The Long Island Serial Killer</td>\n",
+ " <td id=\"T_60df5_row1_col3\" class=\"data row1 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_60df5_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
+ " <td id=\"T_60df5_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
+ " <td id=\"T_60df5_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
+ " <td id=\"T_60df5_row2_col2\" class=\"data row2 col2\" >Melbourne, Australia</td>\n",
+ " <td id=\"T_60df5_row2_col3\" class=\"data row2 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_60df5_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
+ " <td id=\"T_60df5_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
+ " <td id=\"T_60df5_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
+ " <td id=\"T_60df5_row3_col2\" class=\"data row3 col2\" >1966</td>\n",
+ " <td id=\"T_60df5_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_60df5_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
+ " <td id=\"T_60df5_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
+ " <td id=\"T_60df5_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
+ " <td id=\"T_60df5_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
+ " <td id=\"T_60df5_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_60df5_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
+ " <td id=\"T_60df5_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
+ " <td id=\"T_60df5_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
+ " <td id=\"T_60df5_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
+ " <td id=\"T_60df5_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_60df5_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
+ " <td id=\"T_60df5_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
+ " <td id=\"T_60df5_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
+ " <td id=\"T_60df5_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
+ " <td id=\"T_60df5_row6_col3\" class=\"data row6 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_60df5_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
+ " <td id=\"T_60df5_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
+ " <td id=\"T_60df5_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
+ " <td id=\"T_60df5_row7_col2\" class=\"data row7 col2\" >July 30, 1953</td>\n",
+ " <td id=\"T_60df5_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_60df5_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
+ " <td id=\"T_60df5_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
+ " <td id=\"T_60df5_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
+ " <td id=\"T_60df5_row8_col2\" class=\"data row8 col2\" >Hudson River</td>\n",
+ " <td id=\"T_60df5_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_60df5_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
+ " <td id=\"T_60df5_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
+ " <td id=\"T_60df5_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
+ " <td id=\"T_60df5_row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",
+ " <td id=\"T_60df5_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_60df5_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
+ " <td id=\"T_60df5_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
+ " <td id=\"T_60df5_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
+ " <td id=\"T_60df5_row10_col2\" class=\"data row10 col2\" >Hong Kong</td>\n",
+ " <td id=\"T_60df5_row10_col3\" class=\"data row10 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_60df5_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
+ " <td id=\"T_60df5_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
+ " <td id=\"T_60df5_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
+ " <td id=\"T_60df5_row11_col2\" class=\"data row11 col2\" >Venus</td>\n",
+ " <td id=\"T_60df5_row11_col3\" class=\"data row11 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_60df5_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
+ " <td id=\"T_60df5_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
+ " <td id=\"T_60df5_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
+ " <td id=\"T_60df5_row12_col2\" class=\"data row12 col2\" >1457</td>\n",
+ " <td id=\"T_60df5_row12_col3\" class=\"data row12 col3\" >❌</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n"
+ ],
+ "text/plain": [
+ "<pandas.io.formats.style.Styler at 0x7fac858f92e0>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "23.1"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from dsp.evaluation.utils import evaluate\n",
+ "\n",
+ "evaluate(Direct_Query_QA, dev)"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python (DSP)",
+ "language": "python",
+ "name": "dsp"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.16"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}