DSPy: The framework for programming—not prompting—language models
Python
Update examples/tweets/compiling_langchain.ipynb
Omar Khattab2caefa2e685d
diff --git a/.gitignore b/.gitignoreindex 7716fe28f..c47837feb 100644--- a/.gitignore+++ b/.gitignore@@ -42,4 +42,4 @@ finetuning_ckpts/.ideaassertion.log*.log-cache.db\ No newline at end of file+*.db\ No newline at end of filediff --git a/examples/tweets/compiling_langchain.ipynb b/examples/tweets/compiling_langchain.ipynbindex 4e183c043..e847df323 100644--- a/examples/tweets/compiling_langchain.ipynb+++ b/examples/tweets/compiling_langchain.ipynb@@ -34,7 +34,7 @@"cell_type": "markdown","metadata": {},"source": [- "Typically, we use DSPy optimizers with DSPy modules. But here, we've worked with Harrison Chase to make sure DSPy can also optimize chains built with the `LangChain` library.\n",+ "Typically, we use DSPy optimizers with DSPy modules. But here, we've worked with [Harrison Chase](https://twitter.com/hwchase17) to make sure DSPy can also optimize chains built with the `LangChain` library.\n","\n","This short tutorial demonstrates how this proof-of-concept feature works. _This will **not** give you the full power of DSPy or LangChain yet, but we will expand it if there's high demand._\n","\n",@@ -70,10 +70,9 @@"from dspy.evaluate.evaluate import Evaluate\n","from dspy.teleprompt import BootstrapFewShotWithRandomSearch\n","\n",- "turbo = dspy.OpenAI(model='gpt-3.5-turbo-instruct', max_tokens=500)\n","colbertv2 = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')\n","\n",- "dspy.configure(lm=turbo, rm=colbertv2)"+ "dspy.configure(rm=colbertv2)"]},{@@ -283,7 +282,7 @@"name": "stderr","output_type": "stream","text": [- "Average Metric: 63.999999999999986 / 150 (42.7): 100%|██████████| 150/150 [00:02<00:00, 65.78it/s]\n",+ "Average Metric: 63.999999999999986 / 150 (42.7): 100%|██████████| 150/150 [00:02<00:00, 66.08it/s]\n","/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/dspy/evaluate/evaluate.py:126: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n"," df = df.applymap(truncate_cell)\n"]@@ -299,82 +298,82 @@"data": {"text/html": ["<style type=\"text/css\">\n",- "#T_ed282 th {\n",+ "#T_c5a89 th {\n"," text-align: left;\n","}\n",- "#T_ed282 td {\n",+ "#T_c5a89 td {\n"," text-align: left;\n","}\n",- "#T_ed282_row0_col0, #T_ed282_row0_col1, #T_ed282_row0_col2, #T_ed282_row0_col3, #T_ed282_row0_col4, #T_ed282_row0_col5, #T_ed282_row1_col0, #T_ed282_row1_col1, #T_ed282_row1_col2, #T_ed282_row1_col3, #T_ed282_row1_col4, #T_ed282_row1_col5, #T_ed282_row2_col0, #T_ed282_row2_col1, #T_ed282_row2_col2, #T_ed282_row2_col3, #T_ed282_row2_col4, #T_ed282_row2_col5, #T_ed282_row3_col0, #T_ed282_row3_col1, #T_ed282_row3_col2, #T_ed282_row3_col3, #T_ed282_row3_col4, #T_ed282_row3_col5, #T_ed282_row4_col0, #T_ed282_row4_col1, #T_ed282_row4_col2, #T_ed282_row4_col3, #T_ed282_row4_col4, #T_ed282_row4_col5 {\n",+ "#T_c5a89_row0_col0, #T_c5a89_row0_col1, #T_c5a89_row0_col2, #T_c5a89_row0_col3, #T_c5a89_row0_col4, #T_c5a89_row0_col5, #T_c5a89_row1_col0, #T_c5a89_row1_col1, #T_c5a89_row1_col2, #T_c5a89_row1_col3, #T_c5a89_row1_col4, #T_c5a89_row1_col5, #T_c5a89_row2_col0, #T_c5a89_row2_col1, #T_c5a89_row2_col2, #T_c5a89_row2_col3, #T_c5a89_row2_col4, #T_c5a89_row2_col5, #T_c5a89_row3_col0, #T_c5a89_row3_col1, #T_c5a89_row3_col2, #T_c5a89_row3_col3, #T_c5a89_row3_col4, #T_c5a89_row3_col5, #T_c5a89_row4_col0, #T_c5a89_row4_col1, #T_c5a89_row4_col2, #T_c5a89_row4_col3, #T_c5a89_row4_col4, #T_c5a89_row4_col5 {\n"," text-align: left;\n"," white-space: pre-wrap;\n"," word-wrap: break-word;\n"," max-width: 400px;\n","}\n","</style>\n",- "<table id=\"T_ed282\">\n",+ "<table id=\"T_c5a89\">\n"," <thead>\n"," <tr>\n"," <th class=\"blank level0\" > </th>\n",- " <th id=\"T_ed282_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",- " <th id=\"T_ed282_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",- " <th id=\"T_ed282_level0_col2\" class=\"col_heading level0 col2\" >gold_titles</th>\n",- " <th id=\"T_ed282_level0_col3\" class=\"col_heading level0 col3\" >output</th>\n",- " <th id=\"T_ed282_level0_col4\" class=\"col_heading level0 col4\" >tweet_response</th>\n",- " <th id=\"T_ed282_level0_col5\" class=\"col_heading level0 col5\" >metric</th>\n",+ " <th id=\"T_c5a89_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",+ " <th id=\"T_c5a89_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",+ " <th id=\"T_c5a89_level0_col2\" class=\"col_heading level0 col2\" >gold_titles</th>\n",+ " <th id=\"T_c5a89_level0_col3\" class=\"col_heading level0 col3\" >output</th>\n",+ " <th id=\"T_c5a89_level0_col4\" class=\"col_heading level0 col4\" >tweet_response</th>\n",+ " <th id=\"T_c5a89_level0_col5\" class=\"col_heading level0 col5\" >metric</th>\n"," </tr>\n"," </thead>\n"," <tbody>\n"," <tr>\n",- " <th id=\"T_ed282_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",- " <td id=\"T_ed282_row0_col0\" class=\"data row0 col0\" >Who was a producer who produced albums for both rock bands Juke Karten and Thirty Seconds to Mars?</td>\n",- " <td id=\"T_ed282_row0_col1\" class=\"data row0 col1\" >Brian Virtue</td>\n",- " <td id=\"T_ed282_row0_col2\" class=\"data row0 col2\" >{'Levolution (album)', 'Thirty Seconds to Mars'}</td>\n",- " <td id=\"T_ed282_row0_col3\" class=\"data row0 col3\" >Brian Virtue, who has worked with bands like Jane's Addiction and Velvet Revolver, produced albums for both Juke Kartel and Thirty Seconds to Mars, showcasing...</td>\n",- " <td id=\"T_ed282_row0_col4\" class=\"data row0 col4\" >Brian Virtue, who has worked with bands like Jane's Addiction and Velvet Revolver, produced albums for both Juke Kartel and Thirty Seconds to Mars, showcasing...</td>\n",- " <td id=\"T_ed282_row0_col5\" class=\"data row0 col5\" >1.0</td>\n",+ " <th id=\"T_c5a89_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",+ " <td id=\"T_c5a89_row0_col0\" class=\"data row0 col0\" >Who was a producer who produced albums for both rock bands Juke Karten and Thirty Seconds to Mars?</td>\n",+ " <td id=\"T_c5a89_row0_col1\" class=\"data row0 col1\" >Brian Virtue</td>\n",+ " <td id=\"T_c5a89_row0_col2\" class=\"data row0 col2\" >{'Thirty Seconds to Mars', 'Levolution (album)'}</td>\n",+ " <td id=\"T_c5a89_row0_col3\" class=\"data row0 col3\" >Brian Virtue, who has worked with bands like Jane's Addiction and Velvet Revolver, produced albums for both Juke Kartel and Thirty Seconds to Mars, showcasing...</td>\n",+ " <td id=\"T_c5a89_row0_col4\" class=\"data row0 col4\" >Brian Virtue, who has worked with bands like Jane's Addiction and Velvet Revolver, produced albums for both Juke Kartel and Thirty Seconds to Mars, showcasing...</td>\n",+ " <td id=\"T_c5a89_row0_col5\" class=\"data row0 col5\" >1.0</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_ed282_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",- " <td id=\"T_ed282_row1_col0\" class=\"data row1 col0\" >Are both the University of Chicago and Syracuse University public universities? </td>\n",- " <td id=\"T_ed282_row1_col1\" class=\"data row1 col1\" >no</td>\n",- " <td id=\"T_ed282_row1_col2\" class=\"data row1 col2\" >{'Syracuse University', 'University of Chicago'}</td>\n",- " <td id=\"T_ed282_row1_col3\" class=\"data row1 col3\" > No, only Syracuse University is a public university. The University of Chicago is a private research university. #Syracuse #University #Chicago #Public #Private</td>\n",- " <td id=\"T_ed282_row1_col4\" class=\"data row1 col4\" > No, only Syracuse University is a public university. The University of Chicago is a private research university. #Syracuse #University #Chicago #Public #Private</td>\n",- " <td id=\"T_ed282_row1_col5\" class=\"data row1 col5\" >0.3333333333333333</td>\n",+ " <th id=\"T_c5a89_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",+ " <td id=\"T_c5a89_row1_col0\" class=\"data row1 col0\" >Are both the University of Chicago and Syracuse University public universities? </td>\n",+ " <td id=\"T_c5a89_row1_col1\" class=\"data row1 col1\" >no</td>\n",+ " <td id=\"T_c5a89_row1_col2\" class=\"data row1 col2\" >{'Syracuse University', 'University of Chicago'}</td>\n",+ " <td id=\"T_c5a89_row1_col3\" class=\"data row1 col3\" > No, only Syracuse University is a public university. The University of Chicago is a private research university. #Syracuse #University #Chicago #Public #Private</td>\n",+ " <td id=\"T_c5a89_row1_col4\" class=\"data row1 col4\" > No, only Syracuse University is a public university. The University of Chicago is a private research university. #Syracuse #University #Chicago #Public #Private</td>\n",+ " <td id=\"T_c5a89_row1_col5\" class=\"data row1 col5\" >0.3333333333333333</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_ed282_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",- " <td id=\"T_ed282_row2_col0\" class=\"data row2 col0\" >In what region was Eddy Mazzoleni born?</td>\n",- " <td id=\"T_ed282_row2_col1\" class=\"data row2 col1\" >Lombardy, northern Italy</td>\n",- " <td id=\"T_ed282_row2_col2\" class=\"data row2 col2\" >{'Bergamo', 'Eddy Mazzoleni'}</td>\n",- " <td id=\"T_ed282_row2_col3\" class=\"data row2 col3\" > Eddy Mazzoleni, Italian professional cyclist, was born in Bergamo, Italy on July 29, 1973. #cyclist #Italy #Bergamo</td>\n",- " <td id=\"T_ed282_row2_col4\" class=\"data row2 col4\" > Eddy Mazzoleni, Italian professional cyclist, was born in Bergamo, Italy on July 29, 1973. #cyclist #Italy #Bergamo</td>\n",- " <td id=\"T_ed282_row2_col5\" class=\"data row2 col5\" >0.0</td>\n",+ " <th id=\"T_c5a89_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",+ " <td id=\"T_c5a89_row2_col0\" class=\"data row2 col0\" >In what region was Eddy Mazzoleni born?</td>\n",+ " <td id=\"T_c5a89_row2_col1\" class=\"data row2 col1\" >Lombardy, northern Italy</td>\n",+ " <td id=\"T_c5a89_row2_col2\" class=\"data row2 col2\" >{'Eddy Mazzoleni', 'Bergamo'}</td>\n",+ " <td id=\"T_c5a89_row2_col3\" class=\"data row2 col3\" > Eddy Mazzoleni, Italian professional cyclist, was born in Bergamo, Italy on July 29, 1973. #cyclist #Italy #Bergamo</td>\n",+ " <td id=\"T_c5a89_row2_col4\" class=\"data row2 col4\" > Eddy Mazzoleni, Italian professional cyclist, was born in Bergamo, Italy on July 29, 1973. #cyclist #Italy #Bergamo</td>\n",+ " <td id=\"T_c5a89_row2_col5\" class=\"data row2 col5\" >0.0</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_ed282_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",- " <td id=\"T_ed282_row3_col0\" class=\"data row3 col0\" >Who edited the 1990 American romantic comedy film directed by Garry Marshall?</td>\n",- " <td id=\"T_ed282_row3_col1\" class=\"data row3 col1\" >Raja Raymond Gosnell</td>\n",- " <td id=\"T_ed282_row3_col2\" class=\"data row3 col2\" >{'Raja Gosnell', 'Pretty Woman'}</td>\n",- " <td id=\"T_ed282_row3_col3\" class=\"data row3 col3\" > J. F. Lawton edited the 1990 American romantic comedy film directed by Garry Marshall. #PrettyWoman #GarryMarshall #JFLawton</td>\n",- " <td id=\"T_ed282_row3_col4\" class=\"data row3 col4\" > J. F. Lawton edited the 1990 American romantic comedy film directed by Garry Marshall. #PrettyWoman #GarryMarshall #JFLawton</td>\n",- " <td id=\"T_ed282_row3_col5\" class=\"data row3 col5\" >0.0</td>\n",+ " <th id=\"T_c5a89_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",+ " <td id=\"T_c5a89_row3_col0\" class=\"data row3 col0\" >Who edited the 1990 American romantic comedy film directed by Garry Marshall?</td>\n",+ " <td id=\"T_c5a89_row3_col1\" class=\"data row3 col1\" >Raja Raymond Gosnell</td>\n",+ " <td id=\"T_c5a89_row3_col2\" class=\"data row3 col2\" >{'Raja Gosnell', 'Pretty Woman'}</td>\n",+ " <td id=\"T_c5a89_row3_col3\" class=\"data row3 col3\" > J. F. Lawton edited the 1990 American romantic comedy film directed by Garry Marshall. #PrettyWoman #GarryMarshall #JFLawton</td>\n",+ " <td id=\"T_c5a89_row3_col4\" class=\"data row3 col4\" > J. F. Lawton edited the 1990 American romantic comedy film directed by Garry Marshall. #PrettyWoman #GarryMarshall #JFLawton</td>\n",+ " <td id=\"T_c5a89_row3_col5\" class=\"data row3 col5\" >0.0</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_ed282_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",- " <td id=\"T_ed282_row4_col0\" class=\"data row4 col0\" >Burrs Country Park railway station is what stop on the railway line that runs between Heywood and Rawtenstall</td>\n",- " <td id=\"T_ed282_row4_col1\" class=\"data row4 col1\" >seventh</td>\n",- " <td id=\"T_ed282_row4_col2\" class=\"data row4 col2\" >{'East Lancashire Railway', 'Burrs Country Park railway station'}</td>\n",- " <td id=\"T_ed282_row4_col3\" class=\"data row4 col3\" > Burrs Country Park railway station is the seventh stop on the East Lancashire Railway line that runs between Heywood and Rawtenstall.</td>\n",- " <td id=\"T_ed282_row4_col4\" class=\"data row4 col4\" > Burrs Country Park railway station is the seventh stop on the East Lancashire Railway line that runs between Heywood and Rawtenstall.</td>\n",- " <td id=\"T_ed282_row4_col5\" class=\"data row4 col5\" >1.0</td>\n",+ " <th id=\"T_c5a89_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",+ " <td id=\"T_c5a89_row4_col0\" class=\"data row4 col0\" >Burrs Country Park railway station is what stop on the railway line that runs between Heywood and Rawtenstall</td>\n",+ " <td id=\"T_c5a89_row4_col1\" class=\"data row4 col1\" >seventh</td>\n",+ " <td id=\"T_c5a89_row4_col2\" class=\"data row4 col2\" >{'East Lancashire Railway', 'Burrs Country Park railway station'}</td>\n",+ " <td id=\"T_c5a89_row4_col3\" class=\"data row4 col3\" > Burrs Country Park railway station is the seventh stop on the East Lancashire Railway line that runs between Heywood and Rawtenstall.</td>\n",+ " <td id=\"T_c5a89_row4_col4\" class=\"data row4 col4\" > Burrs Country Park railway station is the seventh stop on the East Lancashire Railway line that runs between Heywood and Rawtenstall.</td>\n",+ " <td id=\"T_c5a89_row4_col5\" class=\"data row4 col5\" >1.0</td>\n"," </tr>\n"," </tbody>\n","</table>\n"],"text/plain": [- "<pandas.io.formats.style.Styler at 0x7f8c58f5f6d0>"+ "<pandas.io.formats.style.Styler at 0x7f0d98743c40>"]},"metadata": {},@@ -460,7 +459,7 @@"name": "stderr","output_type": "stream","text": [- "Average Metric: 22.333333333333336 / 50 (44.7): 100%|██████████| 50/50 [00:00<00:00, 55.76it/s]\n",+ "Average Metric: 22.333333333333336 / 50 (44.7): 100%|██████████| 50/50 [00:00<00:00, 55.47it/s]\n","/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/dspy/evaluate/evaluate.py:126: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n"," df = df.applymap(truncate_cell)\n"]@@ -480,7 +479,7 @@"name": "stderr","output_type": "stream","text": [- "Average Metric: 22.333333333333332 / 50 (44.7): 100%|██████████| 50/50 [00:00<00:00, 224.17it/s]\n",+ "Average Metric: 22.333333333333336 / 50 (44.7): 100%|██████████| 50/50 [00:00<00:00, 166.70it/s]\n","/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/dspy/evaluate/evaluate.py:126: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n"," df = df.applymap(truncate_cell)\n"]@@ -489,7 +488,7 @@"name": "stdout","output_type": "stream","text": [- "Average Metric: 22.333333333333332 / 50 (44.7%)\n",+ "Average Metric: 22.333333333333336 / 50 (44.7%)\n","Score: 44.67 for set: [16]\n","Scores so far: [44.67, 44.67]\n","Best score: 44.67\n"@@ -499,7 +498,7 @@"name": "stderr","output_type": "stream","text": [- " 2%|▎ | 5/200 [00:00<00:07, 27.18it/s]\n"+ " 2%|▎ | 5/200 [00:00<00:07, 26.88it/s]\n"]},{@@ -513,7 +512,7 @@"name": "stderr","output_type": "stream","text": [- "Average Metric: 27.000000000000004 / 50 (54.0): 100%|██████████| 50/50 [00:00<00:00, 71.93it/s]\n",+ "Average Metric: 27.000000000000004 / 50 (54.0): 100%|██████████| 50/50 [00:00<00:00, 72.21it/s]\n","/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/dspy/evaluate/evaluate.py:126: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n"," df = df.applymap(truncate_cell)\n"]@@ -539,7 +538,7 @@"name": "stderr","output_type": "stream","text": [- " 4%|▍ | 9/200 [00:00<00:06, 27.43it/s]\n"+ " 4%|▍ | 9/200 [00:00<00:06, 28.04it/s]\n"]},{@@ -553,7 +552,7 @@"name": "stderr","output_type": "stream","text": [- "Average Metric: 25.000000000000004 / 50 (50.0): 100%|██████████| 50/50 [00:00<00:00, 74.02it/s]\n",+ "Average Metric: 25.000000000000007 / 50 (50.0): 100%|██████████| 50/50 [00:00<00:00, 70.71it/s]\n","/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/dspy/evaluate/evaluate.py:126: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n"," df = df.applymap(truncate_cell)\n"]@@ -562,7 +561,7 @@"name": "stdout","output_type": "stream","text": [- "Average Metric: 25.000000000000004 / 50 (50.0%)\n",+ "Average Metric: 25.000000000000007 / 50 (50.0%)\n","Score: 50.0 for set: [16]\n","Scores so far: [44.67, 44.67, 54.0, 50.0]\n","Best score: 54.0\n",@@ -578,7 +577,7 @@"name": "stderr","output_type": "stream","text": [- " 0%| | 1/200 [00:00<00:07, 27.41it/s]\n"+ " 0%| | 1/200 [00:00<00:07, 28.24it/s]\n"]},{@@ -592,7 +591,7 @@"name": "stderr","output_type": "stream","text": [- "Average Metric: 25.666666666666668 / 50 (51.3): 100%|██████████| 50/50 [00:00<00:00, 78.28it/s]\n",+ "Average Metric: 25.666666666666664 / 50 (51.3): 100%|██████████| 50/50 [00:00<00:00, 75.37it/s]\n","/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/dspy/evaluate/evaluate.py:126: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n"," df = df.applymap(truncate_cell)\n"]@@ -601,7 +600,7 @@"name": "stdout","output_type": "stream","text": [- "Average Metric: 25.666666666666668 / 50 (51.3%)\n",+ "Average Metric: 25.666666666666664 / 50 (51.3%)\n","Score: 51.33 for set: [16]\n","Scores so far: [44.67, 44.67, 54.0, 50.0, 51.33]\n","Best score: 54.0\n",@@ -617,7 +616,7 @@"name": "stderr","output_type": "stream","text": [- " 1%| | 2/200 [00:00<00:07, 26.31it/s]\n"+ " 1%| | 2/200 [00:00<00:07, 27.81it/s]\n"]},{@@ -631,14 +630,16 @@"name": "stderr","output_type": "stream","text": [- "Average Metric: 26.000000000000004 / 50 (52.0): 100%|██████████| 50/50 [00:00<00:00, 74.75it/s]"+ "Average Metric: 26.0 / 50 (52.0): 100%|██████████| 50/50 [00:00<00:00, 73.67it/s] \n",+ "/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/dspy/evaluate/evaluate.py:126: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n",+ " df = df.applymap(truncate_cell)\n"]},{"name": "stdout","output_type": "stream","text": [- "Average Metric: 26.000000000000004 / 50 (52.0%)\n",+ "Average Metric: 26.0 / 50 (52.0%)\n","Score: 52.0 for set: [16]\n","Scores so far: [44.67, 44.67, 54.0, 50.0, 51.33, 52.0]\n","Best score: 54.0\n",@@ -650,15 +651,6 @@"Average of max per entry across top 9999 scores: 0.64\n","6 candidate programs found.\n"]- },- {- "name": "stderr",- "output_type": "stream",- "text": [- "\n",- "/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/dspy/evaluate/evaluate.py:126: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n",- " df = df.applymap(truncate_cell)\n"- ]}],"source": [@@ -719,7 +711,7 @@"name": "stderr","output_type": "stream","text": [- "Average Metric: 78.66666666666667 / 150 (52.4): 100%|██████████| 150/150 [00:02<00:00, 71.07it/s] "+ "Average Metric: 78.66666666666667 / 150 (52.4): 100%|██████████| 150/150 [00:02<00:00, 72.64it/s] "]},{@@ -742,82 +734,82 @@"data": {"text/html": ["<style type=\"text/css\">\n",- "#T_c3dbe th {\n",+ "#T_dddf8 th {\n"," text-align: left;\n","}\n",- "#T_c3dbe td {\n",+ "#T_dddf8 td {\n"," text-align: left;\n","}\n",- "#T_c3dbe_row0_col0, #T_c3dbe_row0_col1, #T_c3dbe_row0_col2, #T_c3dbe_row0_col3, #T_c3dbe_row0_col4, #T_c3dbe_row0_col5, #T_c3dbe_row1_col0, #T_c3dbe_row1_col1, #T_c3dbe_row1_col2, #T_c3dbe_row1_col3, #T_c3dbe_row1_col4, #T_c3dbe_row1_col5, #T_c3dbe_row2_col0, #T_c3dbe_row2_col1, #T_c3dbe_row2_col2, #T_c3dbe_row2_col3, #T_c3dbe_row2_col4, #T_c3dbe_row2_col5, #T_c3dbe_row3_col0, #T_c3dbe_row3_col1, #T_c3dbe_row3_col2, #T_c3dbe_row3_col3, #T_c3dbe_row3_col4, #T_c3dbe_row3_col5, #T_c3dbe_row4_col0, #T_c3dbe_row4_col1, #T_c3dbe_row4_col2, #T_c3dbe_row4_col3, #T_c3dbe_row4_col4, #T_c3dbe_row4_col5 {\n",+ "#T_dddf8_row0_col0, #T_dddf8_row0_col1, #T_dddf8_row0_col2, #T_dddf8_row0_col3, #T_dddf8_row0_col4, #T_dddf8_row0_col5, #T_dddf8_row1_col0, #T_dddf8_row1_col1, #T_dddf8_row1_col2, #T_dddf8_row1_col3, #T_dddf8_row1_col4, #T_dddf8_row1_col5, #T_dddf8_row2_col0, #T_dddf8_row2_col1, #T_dddf8_row2_col2, #T_dddf8_row2_col3, #T_dddf8_row2_col4, #T_dddf8_row2_col5, #T_dddf8_row3_col0, #T_dddf8_row3_col1, #T_dddf8_row3_col2, #T_dddf8_row3_col3, #T_dddf8_row3_col4, #T_dddf8_row3_col5, #T_dddf8_row4_col0, #T_dddf8_row4_col1, #T_dddf8_row4_col2, #T_dddf8_row4_col3, #T_dddf8_row4_col4, #T_dddf8_row4_col5 {\n"," text-align: left;\n"," white-space: pre-wrap;\n"," word-wrap: break-word;\n"," max-width: 400px;\n","}\n","</style>\n",- "<table id=\"T_c3dbe\">\n",+ "<table id=\"T_dddf8\">\n"," <thead>\n"," <tr>\n"," <th class=\"blank level0\" > </th>\n",- " <th id=\"T_c3dbe_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",- " <th id=\"T_c3dbe_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",- " <th id=\"T_c3dbe_level0_col2\" class=\"col_heading level0 col2\" >gold_titles</th>\n",- " <th id=\"T_c3dbe_level0_col3\" class=\"col_heading level0 col3\" >output</th>\n",- " <th id=\"T_c3dbe_level0_col4\" class=\"col_heading level0 col4\" >tweet_response</th>\n",- " <th id=\"T_c3dbe_level0_col5\" class=\"col_heading level0 col5\" >metric</th>\n",+ " <th id=\"T_dddf8_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",+ " <th id=\"T_dddf8_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",+ " <th id=\"T_dddf8_level0_col2\" class=\"col_heading level0 col2\" >gold_titles</th>\n",+ " <th id=\"T_dddf8_level0_col3\" class=\"col_heading level0 col3\" >output</th>\n",+ " <th id=\"T_dddf8_level0_col4\" class=\"col_heading level0 col4\" >tweet_response</th>\n",+ " <th id=\"T_dddf8_level0_col5\" class=\"col_heading level0 col5\" >metric</th>\n"," </tr>\n"," </thead>\n"," <tbody>\n"," <tr>\n",- " <th id=\"T_c3dbe_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",- " <td id=\"T_c3dbe_row0_col0\" class=\"data row0 col0\" >Who was a producer who produced albums for both rock bands Juke Karten and Thirty Seconds to Mars?</td>\n",- " <td id=\"T_c3dbe_row0_col1\" class=\"data row0 col1\" >Brian Virtue</td>\n",- " <td id=\"T_c3dbe_row0_col2\" class=\"data row0 col2\" >{'Levolution (album)', 'Thirty Seconds to Mars'}</td>\n",- " <td id=\"T_c3dbe_row0_col3\" class=\"data row0 col3\" >Brian Virtue is a producer who has worked with both Juke Kartel and Thirty Seconds to Mars, helping to create their unique sounds. #BrianVirtue #producer...</td>\n",- " <td id=\"T_c3dbe_row0_col4\" class=\"data row0 col4\" >Brian Virtue is a producer who has worked with both Juke Kartel and Thirty Seconds to Mars, helping to create their unique sounds. #BrianVirtue #producer...</td>\n",- " <td id=\"T_c3dbe_row0_col5\" class=\"data row0 col5\" >1.0</td>\n",+ " <th id=\"T_dddf8_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",+ " <td id=\"T_dddf8_row0_col0\" class=\"data row0 col0\" >Who was a producer who produced albums for both rock bands Juke Karten and Thirty Seconds to Mars?</td>\n",+ " <td id=\"T_dddf8_row0_col1\" class=\"data row0 col1\" >Brian Virtue</td>\n",+ " <td id=\"T_dddf8_row0_col2\" class=\"data row0 col2\" >{'Thirty Seconds to Mars', 'Levolution (album)'}</td>\n",+ " <td id=\"T_dddf8_row0_col3\" class=\"data row0 col3\" >Brian Virtue is a producer who has worked with both Juke Kartel and Thirty Seconds to Mars, helping to create their unique sounds. #BrianVirtue #producer...</td>\n",+ " <td id=\"T_dddf8_row0_col4\" class=\"data row0 col4\" >Brian Virtue is a producer who has worked with both Juke Kartel and Thirty Seconds to Mars, helping to create their unique sounds. #BrianVirtue #producer...</td>\n",+ " <td id=\"T_dddf8_row0_col5\" class=\"data row0 col5\" >1.0</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_c3dbe_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",- " <td id=\"T_c3dbe_row1_col0\" class=\"data row1 col0\" >Are both the University of Chicago and Syracuse University public universities? </td>\n",- " <td id=\"T_c3dbe_row1_col1\" class=\"data row1 col1\" >no</td>\n",- " <td id=\"T_c3dbe_row1_col2\" class=\"data row1 col2\" >{'Syracuse University', 'University of Chicago'}</td>\n",- " <td id=\"T_c3dbe_row1_col3\" class=\"data row1 col3\" > Yes, both Northeastern Illinois University and Syracuse University are public universities. #publicuniversity #Chicago #Syracuse</td>\n",- " <td id=\"T_c3dbe_row1_col4\" class=\"data row1 col4\" > Yes, both Northeastern Illinois University and Syracuse University are public universities. #publicuniversity #Chicago #Syracuse</td>\n",- " <td id=\"T_c3dbe_row1_col5\" class=\"data row1 col5\" >0.0</td>\n",+ " <th id=\"T_dddf8_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",+ " <td id=\"T_dddf8_row1_col0\" class=\"data row1 col0\" >Are both the University of Chicago and Syracuse University public universities? </td>\n",+ " <td id=\"T_dddf8_row1_col1\" class=\"data row1 col1\" >no</td>\n",+ " <td id=\"T_dddf8_row1_col2\" class=\"data row1 col2\" >{'Syracuse University', 'University of Chicago'}</td>\n",+ " <td id=\"T_dddf8_row1_col3\" class=\"data row1 col3\" > Yes, both Northeastern Illinois University and Syracuse University are public universities. #publicuniversity #Chicago #Syracuse</td>\n",+ " <td id=\"T_dddf8_row1_col4\" class=\"data row1 col4\" > Yes, both Northeastern Illinois University and Syracuse University are public universities. #publicuniversity #Chicago #Syracuse</td>\n",+ " <td id=\"T_dddf8_row1_col5\" class=\"data row1 col5\" >0.0</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_c3dbe_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",- " <td id=\"T_c3dbe_row2_col0\" class=\"data row2 col0\" >In what region was Eddy Mazzoleni born?</td>\n",- " <td id=\"T_c3dbe_row2_col1\" class=\"data row2 col1\" >Lombardy, northern Italy</td>\n",- " <td id=\"T_c3dbe_row2_col2\" class=\"data row2 col2\" >{'Bergamo', 'Eddy Mazzoleni'}</td>\n",- " <td id=\"T_c3dbe_row2_col3\" class=\"data row2 col3\" > Eddy Mazzoleni was born in Bergamo, a city in the Lombardy region of Italy. #EddyMazzoleni #Italy #Lombardy</td>\n",- " <td id=\"T_c3dbe_row2_col4\" class=\"data row2 col4\" > Eddy Mazzoleni was born in Bergamo, a city in the Lombardy region of Italy. #EddyMazzoleni #Italy #Lombardy</td>\n",- " <td id=\"T_c3dbe_row2_col5\" class=\"data row2 col5\" >1.0</td>\n",+ " <th id=\"T_dddf8_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",+ " <td id=\"T_dddf8_row2_col0\" class=\"data row2 col0\" >In what region was Eddy Mazzoleni born?</td>\n",+ " <td id=\"T_dddf8_row2_col1\" class=\"data row2 col1\" >Lombardy, northern Italy</td>\n",+ " <td id=\"T_dddf8_row2_col2\" class=\"data row2 col2\" >{'Eddy Mazzoleni', 'Bergamo'}</td>\n",+ " <td id=\"T_dddf8_row2_col3\" class=\"data row2 col3\" > Eddy Mazzoleni was born in Bergamo, a city in the Lombardy region of Italy. #EddyMazzoleni #Italy #Lombardy</td>\n",+ " <td id=\"T_dddf8_row2_col4\" class=\"data row2 col4\" > Eddy Mazzoleni was born in Bergamo, a city in the Lombardy region of Italy. #EddyMazzoleni #Italy #Lombardy</td>\n",+ " <td id=\"T_dddf8_row2_col5\" class=\"data row2 col5\" >1.0</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_c3dbe_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",- " <td id=\"T_c3dbe_row3_col0\" class=\"data row3 col0\" >Who edited the 1990 American romantic comedy film directed by Garry Marshall?</td>\n",- " <td id=\"T_c3dbe_row3_col1\" class=\"data row3 col1\" >Raja Raymond Gosnell</td>\n",- " <td id=\"T_c3dbe_row3_col2\" class=\"data row3 col2\" >{'Raja Gosnell', 'Pretty Woman'}</td>\n",- " <td id=\"T_c3dbe_row3_col3\" class=\"data row3 col3\" > Garry Marshall directed and edited the 1990 American romantic comedy film \"Pretty Woman\", starring Richard Gere and Julia Roberts. #PrettyWoman #GarryMarshall #RomanticComedy</td>\n",- " <td id=\"T_c3dbe_row3_col4\" class=\"data row3 col4\" > Garry Marshall directed and edited the 1990 American romantic comedy film \"Pretty Woman\", starring Richard Gere and Julia Roberts. #PrettyWoman #GarryMarshall #RomanticComedy</td>\n",- " <td id=\"T_c3dbe_row3_col5\" class=\"data row3 col5\" >0.0</td>\n",+ " <th id=\"T_dddf8_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",+ " <td id=\"T_dddf8_row3_col0\" class=\"data row3 col0\" >Who edited the 1990 American romantic comedy film directed by Garry Marshall?</td>\n",+ " <td id=\"T_dddf8_row3_col1\" class=\"data row3 col1\" >Raja Raymond Gosnell</td>\n",+ " <td id=\"T_dddf8_row3_col2\" class=\"data row3 col2\" >{'Raja Gosnell', 'Pretty Woman'}</td>\n",+ " <td id=\"T_dddf8_row3_col3\" class=\"data row3 col3\" > Garry Marshall directed and edited the 1990 American romantic comedy film \"Pretty Woman\", starring Richard Gere and Julia Roberts. #PrettyWoman #GarryMarshall #RomanticComedy</td>\n",+ " <td id=\"T_dddf8_row3_col4\" class=\"data row3 col4\" > Garry Marshall directed and edited the 1990 American romantic comedy film \"Pretty Woman\", starring Richard Gere and Julia Roberts. #PrettyWoman #GarryMarshall #RomanticComedy</td>\n",+ " <td id=\"T_dddf8_row3_col5\" class=\"data row3 col5\" >0.0</td>\n"," </tr>\n"," <tr>\n",- " <th id=\"T_c3dbe_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",- " <td id=\"T_c3dbe_row4_col0\" class=\"data row4 col0\" >Burrs Country Park railway station is what stop on the railway line that runs between Heywood and Rawtenstall</td>\n",- " <td id=\"T_c3dbe_row4_col1\" class=\"data row4 col1\" >seventh</td>\n",- " <td id=\"T_c3dbe_row4_col2\" class=\"data row4 col2\" >{'East Lancashire Railway', 'Burrs Country Park railway station'}</td>\n",- " <td id=\"T_c3dbe_row4_col3\" class=\"data row4 col3\" > Burrs Country Park railway station is the seventh stop on the East Lancashire Railway line, which runs between Heywood and Rawtenstall. #EastLancashireRailway #BurrsCountryPark #railwaystation</td>\n",- " <td id=\"T_c3dbe_row4_col4\" class=\"data row4 col4\" > Burrs Country Park railway station is the seventh stop on the East Lancashire Railway line, which runs between Heywood and Rawtenstall. #EastLancashireRailway #BurrsCountryPark #railwaystation</td>\n",- " <td id=\"T_c3dbe_row4_col5\" class=\"data row4 col5\" >1.0</td>\n",+ " <th id=\"T_dddf8_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",+ " <td id=\"T_dddf8_row4_col0\" class=\"data row4 col0\" >Burrs Country Park railway station is what stop on the railway line that runs between Heywood and Rawtenstall</td>\n",+ " <td id=\"T_dddf8_row4_col1\" class=\"data row4 col1\" >seventh</td>\n",+ " <td id=\"T_dddf8_row4_col2\" class=\"data row4 col2\" >{'East Lancashire Railway', 'Burrs Country Park railway station'}</td>\n",+ " <td id=\"T_dddf8_row4_col3\" class=\"data row4 col3\" > Burrs Country Park railway station is the seventh stop on the East Lancashire Railway line, which runs between Heywood and Rawtenstall. #EastLancashireRailway #BurrsCountryPark #railwaystation</td>\n",+ " <td id=\"T_dddf8_row4_col4\" class=\"data row4 col4\" > Burrs Country Park railway station is the seventh stop on the East Lancashire Railway line, which runs between Heywood and Rawtenstall. #EastLancashireRailway #BurrsCountryPark #railwaystation</td>\n",+ " <td id=\"T_dddf8_row4_col5\" class=\"data row4 col5\" >1.0</td>\n"," </tr>\n"," </tbody>\n","</table>\n"],"text/plain": [- "<pandas.io.formats.style.Styler at 0x7f8c608f90d0>"+ "<pandas.io.formats.style.Styler at 0x7f0d9ab1cac0>"]},"metadata": {},@@ -930,29 +922,38 @@"---\n","\n","Context:\n",- "[1] «Philip Markoff | Philip Haynes Markoff (February 12, 1986 – August 15, 2010) was an American medical student who was charged with the armed robbery and murder of Julissa Brisman in a Boston, Massachusetts, hotel on April 14, 2009, and two other armed robberies.»\n",- "[2] «Antonia Brenner | Antonia Brenner, better known as Mother Antonia (Spanish: Madre Antonia ), (December 1, 1926 – October 17, 2013) was an American Roman Catholic Religious Sister and activist who chose to reside and care for inmates at the notorious maximum-security La Mesa Prison in Tijuana, Mexico. As a result of her work, she founded a new religious institute called the Eudist Servants of the 11th Hour.»\n",- "[3] «Luzira Maximum Security Prison | Luzira Maximum Security Prison is a maximum security prison for both men and women in Uganda. As at July 2016, it is the only maximum security prison in the country and houses Uganda's death row inmates.»\n",- "[4] «Pleasant Valley State Prison | Pleasant Valley State Prison (PVSP) is a 640 acres minimum-to-maximum security state prison in Coalinga, Fresno County, California. The facility has housed convicted murderers Sirhan Sirhan, Erik Menendez, X-Raided, and Hans Reiser, among others.»\n",- "[5] «Jon-Adrian Velazquez | Jon-Adrian Velazquez is an inmate in the maximum security Sing-Sing prison in New York who is serving a 25-year sentence after being convicted of the 1998 murder of a retired police officer. His case garnered considerable attention from the media ten years after his conviction, due to a visit and support from Martin Sheen and a long-term investigation by Dateline NBC producer Dan Slepian.»\n",- "Question: Which maximum security jail housed the killer of Julissa brisman?\n",+ "[1] «Scott Lowell | Scott Lowell (born February 22, 1965 in Denver, Colorado) is an American actor best known for his role as Ted Schmidt on the Showtime drama \"Queer as Folk\".»\n",+ "[2] «Ted Schmidt | Theodore \"Ted\" Schmidt is a fictional character from the American Showtime television drama series \"Queer as Folk\", played by Scott Lowell. Fellow show cast member Peter Paige, who plays Emmett Honeycutt originally auditioned for the role. Lowell was cast and he stated that he had an instant connection with the character. \"Queer as Folk\" is based on the British show of the same name and Ted is loosely based on the character Phil Delaney, played by Jason Merrells. Phil was killed off in that series, whereas show creator Daniel Lipman decided to develop the character into a full-time role for the US version.»\n",+ "[3] «Chris Lowell | Christopher Lowell (born October 17, 1984) is an American television actor. He played the role of Stosh \"Piz\" Piznarski in the CW noir drama \"Veronica Mars\" and the character William \"Dell\" Parker in the ABC \"Grey's Anatomy\" spin-off \"Private Practice\".»\n",+ "[4] «Kevin Schmidt | Kevin Gerard Schmidt (born August 16, 1988) is an American actor, known best for his role as Henry in \"Cheaper by the Dozen\" and its sequel and as Noah Newman in \"The Young and the Restless\". Schmidt also starred on Cartoon Network's first live-action scripted television series, \"Unnatural History\". Schmidt also co-created, starred in, produced, and directed a cult web-series, \"Poor Paul\". Schmidt continues to write, direct, and act, and has also participated in humanitarian organizations. Schmidt is president of the Conscious Human Initiative, a non-profit entity that intends to alleviate malnutrition worldwide. He played Ryan in .»\n",+ "[5] «Frederick Koehler | Frederick Koehler (born June 16, 1975) is an American actor best known for his role as Chip Lowell on \"Kate & Allie\" as well as Andrew Schillinger on the HBO drama \"Oz\". He is distinguished for appearing much younger than his chronological age (e.g., appearing about 20 years old when he was actually 38).»\n",+ "Question: What show is an American-Canadian drama starring Scott Lowell playing Ted Schmidt?\n","Tweet Response:\n","\n","\n","OUTPUT:\n","\n"," Prediction(\n",- " tweet_response=' Philip Markoff, the killer of Julissa Brisman, was housed in the maximum security jail Pleasant Valley State Prison in California. #PhilipMarkoff #PleasantValleyStatePrison #maximumsecurity'\n",+ " tweet_response=' Scott Lowell played Ted Schmidt on the American-Canadian drama \"Queer as Folk\". #ScottLowell #TedSchmidt #QueerAsFolk'\n",")\n"]}],"source": [- "prompt, output = dspy.settings.langchain_history[-1]\n",+ "prompt, output = dspy.settings.langchain_history[-4]\n","\n","print('PROMPT:\\n\\n', prompt)\n","print('\\n\\nOUTPUT:\\n\\n', output)"]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "#### Acknowledgements:\n",+ "\n",+ "Thanks to [Harrison Chase](https://twitter.com/hwchase17) for co-leading this new integration. Thanks to our own [Arnav Singhvi](https://arnavsinghvi11.github.io/) for helping cook this tweet generation task and the insight about how to get data to use here."+ ]}],"metadata": {