DSPy: The framework for programming—not prompting—language models
Python
DSPy optimizing LC chains tutorial
Omar Khattabb107f40e76ad
diff --git a/.gitignore b/.gitignoreindex a5442dc44..7716fe28f 100644--- a/.gitignore+++ b/.gitignore@@ -42,3 +42,4 @@ finetuning_ckpts/.ideaassertion.log*.log+cache.db\ No newline at end of filediff --git a/dsp/utils/settings.py b/dsp/utils/settings.pyindex 60575720b..228a960a2 100644--- a/dsp/utils/settings.py+++ b/dsp/utils/settings.py@@ -39,6 +39,7 @@ class Settings(object):release=0,bypass_assert=False,bypass_suggest=False,+ langchain_history=[])cls._instance.__append(config)diff --git a/dspy/evaluate/evaluate.py b/dspy/evaluate/evaluate.pyindex bda8a89c8..476976b2a 100644--- a/dspy/evaluate/evaluate.py+++ b/dspy/evaluate/evaluate.py@@ -188,7 +188,7 @@ def truncate_cell(content):def configure_dataframe_display(df, metric_name):"""Set various pandas display options for DataFrame."""pd.options.display.max_colwidth = None- pd.set_option('display.max_colwidth', 15) # Adjust the number as needed+ pd.set_option('display.max_colwidth', 20) # Adjust the number as neededpd.set_option('display.width', 400) # Adjust# df[metric_name] = df[metric_name].apply(lambda x: f'✔️ [{x}]' if x is True else f'❌ [{x}]')diff --git a/dspy/predict/langchain.py b/dspy/predict/langchain.pyindex 177988876..e3ddd37ce 100644--- a/dspy/predict/langchain.py+++ b/dspy/predict/langchain.py@@ -101,6 +101,7 @@ class LangChainPredict(Predict, Runnable): #, RunnableBinding):# print('#> len(demos) =', len(demos))# print(f"#> {prompt}")# print(f"#> PRED = {content}\n\n\n")+ dspy.settings.langchain_history.append((prompt, pred))if dsp.settings.trace is not None:trace = dsp.settings.tracediff --git a/examples/tweets/compiling_langchain.ipynb b/examples/tweets/compiling_langchain.ipynbnew file mode 100644index 000000000..17ac8b5ca--- /dev/null+++ b/examples/tweets/compiling_langchain.ipynb@@ -0,0 +1,923 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "<img src=\"../../docs/images/DSPy8.png\" alt=\"DSPy7 Image\" height=\"150\"/>\n",+ "\n",+ "## DSPy: Compiling chains from `LangChain`"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "One of the most powerful features in **DSPy** are its powerful optimizers. **DSPy optimizers** can take any LM system and tune the prompts (or the LM weights) to maximize any objective.\n",+ "\n",+ "Optimizers can improve the quality of your LM systems and make your code adaptive to new LMs or new data. This is meant to bring structure and modularity in place of hacky things like (i) manual prompt engineering, (ii) designing complex pipelines for generating synthetic data, (iii) or designing complex pipelines for finetuning."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "# Install the dependencies if needed.\n",+ "# %pip install -U dspy-ai\n",+ "# %pip install -U openai jinja2\n",+ "# %pip install -U langchain langchain-community langchain-openai langchain-core"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Typically, we use DSPy optimizers with DSPy modules. But here, we've worked with Harrison Chase to make sure DSPy can also optimize chains built with the `LangChain` library.\n",+ "\n",+ "This short tutorial demonstrates how this proof-of-concept feature works. _This will **not** give you the full power of DSPy or LangChain yet, but we will expand it if there's high demand._\n",+ "\n",+ "If we convert this into a fuller integration, all users stand to benefit. LangChain users will gain the ability to optimize any chain with any DSPy optimizer. DSPy users will gain the ability to _export_ any DSPy program into an LCEL that supports streaming and tracing, and other rich production-targeted features in LangChain."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 1) Setting Up\n",+ "\n",+ "First, let's import `dspy` and configure the default language model and retrieval model in it."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 2,+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",+ " from .autonotebook import tqdm as notebook_tqdm\n"+ ]+ }+ ],+ "source": [+ "import dspy\n",+ "\n",+ "from dspy.evaluate.evaluate import Evaluate\n",+ "from dspy.teleprompt import BootstrapFewShotWithRandomSearch\n",+ "\n",+ "turbo = dspy.OpenAI(model='gpt-3.5-turbo-instruct', max_tokens=500)\n",+ "colbertv2 = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')\n",+ "\n",+ "dspy.configure(lm=turbo, rm=colbertv2)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Next, let's import `langchain` and the DSPy modules for interacting with LangChain runnables, namely, `LangChainPredict` and `LangChainModule`."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 3,+ "metadata": {},+ "outputs": [],+ "source": [+ "from langchain_openai import OpenAI\n",+ "from langchain.globals import set_llm_cache\n",+ "from langchain.cache import SQLiteCache\n",+ "\n",+ "set_llm_cache(SQLiteCache(database_path=\"cache.db\"))\n",+ "\n",+ "llm = OpenAI(model_name=\"gpt-3.5-turbo-instruct\", temperature=0)\n",+ "retrieve = lambda x: dspy.Retrieve(k=5)(x[\"question\"]).passages"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "If it's useful, we can set up some caches so you can run this whole notebook in Google Colab without any API keys. Let us know."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 2) Defining a chain as a `LangChain` expression\n",+ "\n",+ "For illustration, let's tackle the following task.\n",+ "\n",+ "**Task:** Build a RAG system for generating informative tweets.\n",+ "- **Input:** A factual **question**, which may be fairly complex.\n",+ "- **Output:** An engaging **tweet** that correctly answers the question from the retrieved info.\n",+ "\n",+ "Let's use LangChain's expression langugage (LCEL) to illustrate this. Any prompt here will do, we will optimize the final prompt with DSPy.\n",+ "\n",+ "Considering that, let's just keep it to the barebones: **Given {context}, answer the question {question} as a tweet.**"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 4,+ "metadata": {},+ "outputs": [],+ "source": [+ "# From LangChain, import standard modules for prompting.\n",+ "from langchain_core.prompts import PromptTemplate\n",+ "from langchain_core.output_parsers import StrOutputParser\n",+ "from langchain_core.runnables import RunnablePassthrough\n",+ "\n",+ "# Just a simple prompt for this task. It's fine if it's complex too.\n",+ "prompt = PromptTemplate.from_template(\"Given {context}, answer the question `{question}` as a tweet.\")\n",+ "\n",+ "# This is how you'd normally build a chain with LCEL. This chain does retrieval then generation (RAG).\n",+ "vanilla_chain = RunnablePassthrough.assign(context=retrieve) | prompt | llm | StrOutputParser()"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 3) Converting the chain into a **DSPy module**"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Our goal is to optimize this prompt so we have a better tweet generator. DSPy optimizers can help, but they only work with DSPy modules!\n",+ "\n",+ "For this reason, we created two new modules in DSPy: `LangChainPredict` and `LangChainModule`."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 5,+ "metadata": {},+ "outputs": [],+ "source": [+ "# From DSPy, import the modules that know how to interact with LangChain LCEL.\n",+ "from dspy.predict.langchain import LangChainPredict, LangChainModule\n",+ "\n",+ "# This is how to wrap it so it behaves like a DSPy program.\n",+ "# Just Replace every pattern like `prompt | llm` with `LangChainPredict(prompt, llm)`.\n",+ "zeroshot_chain = RunnablePassthrough.assign(context=retrieve) | LangChainPredict(prompt, llm) | StrOutputParser()\n",+ "zeroshot_chain = LangChainModule(zeroshot_chain) # then wrap the chain in a DSPy module."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 4) Trying the module\n",+ "\n",+ "How good is our `LangChainModule` at this task? Well, we can ask it to generate a tweet for the following question."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 6,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "' Eddy Mazzoleni, Italian professional cyclist, was born in Bergamo, Italy on July 29, 1973. #cyclist #Italy #Bergamo'"+ ]+ },+ "execution_count": 6,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "question = \"In what region was Eddy Mazzoleni born?\"\n",+ "\n",+ "zeroshot_chain.invoke({\"question\": question})"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Ah that sounds about right! (It's technically not perfect: we asked for the _region_ not the city. We can do better below.)\n",+ "\n",+ "Inspecting questions and answers manually is very important to get a sense of your system. However, a good system designer always looks to iteratively **benchmark** their work to quantify progress!\n",+ "\n",+ "To do this, we need two things: the **metric** we want to maximize and a (tiny) **dataset** of examples for our system.\n",+ "\n",+ "Are there pre-defined metrics for good tweets? Should I label 100,000 tweets by hand? Probably not. We can easily do something reasonable, though, until you start getting data in production!"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 5) Evaluating the module\n",+ "\n",+ "To get started, we'll define our own simple metric and we'll borrow a bunch of questions from a QA dataset and use them here for tuning.\n",+ "\n",+ "**What makes a good tweet?** I don't know, but in the spirit of iterative development, let's start simple!\n",+ "\n",+ "Define a good tweet to be have three properties: it should be (1) factually correct, (2) based on real sources, and (3) engaging for people."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 7,+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/datasets/table.py:1421: FutureWarning: promote has been superseded by mode='default'.\n",+ " table = cls._concat_blocks(blocks, axis=0)\n"+ ]+ },+ {+ "data": {+ "text/plain": [+ "(200, 50, 150)"+ ]+ },+ "execution_count": 7,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "# We took the liberty to define this metric and load a few examples from a standard QA dataset.\n",+ "# Let's impore them from `tweet_metric.py` in the same directory that contains this notebook.\n",+ "from tweet_metric import metric, trainset, valset, devset\n",+ "\n",+ "# We loaded 200, 50, and 150 examples for training, validation (tuning), and development (evaluation), respectively.\n",+ "# You could load less (or more) and, chances are, the right DSPy optimizers will work well for many problems.\n",+ "len(trainset), len(valset), len(devset)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Is this the right metric or the most representative set of questions? Not necessarily. But they get us started in a way we can iterate on systematically!\n",+ "\n",+ "**Note:** Notice that our dataset doesn't actually include any tweets! It only has questions and answers. That's OK, our metric will take care of evaluating outputs in tweet form.\n",+ "\n",+ "Okay, let's evaluate the unoptimized \"zero-shot\" version of our chain, converted from our `LangChain` LCEL object."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 8,+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Average Metric: 63.999999999999986 / 150 (42.7): 100%|██████████| 150/150 [00:02<00:00, 65.97it/s]\n",+ "/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/dspy/evaluate/evaluate.py:126: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n",+ " df = df.applymap(truncate_cell)\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Average Metric: 63.999999999999986 / 150 (42.7%)\n"+ ]+ },+ {+ "data": {+ "text/html": [+ "<style type=\"text/css\">\n",+ "#T_042b5 th {\n",+ " text-align: left;\n",+ "}\n",+ "#T_042b5 td {\n",+ " text-align: left;\n",+ "}\n",+ "#T_042b5_row0_col0, #T_042b5_row0_col1, #T_042b5_row0_col2, #T_042b5_row0_col3, #T_042b5_row0_col4, #T_042b5_row0_col5, #T_042b5_row1_col0, #T_042b5_row1_col1, #T_042b5_row1_col2, #T_042b5_row1_col3, #T_042b5_row1_col4, #T_042b5_row1_col5, #T_042b5_row2_col0, #T_042b5_row2_col1, #T_042b5_row2_col2, #T_042b5_row2_col3, #T_042b5_row2_col4, #T_042b5_row2_col5, #T_042b5_row3_col0, #T_042b5_row3_col1, #T_042b5_row3_col2, #T_042b5_row3_col3, #T_042b5_row3_col4, #T_042b5_row3_col5, #T_042b5_row4_col0, #T_042b5_row4_col1, #T_042b5_row4_col2, #T_042b5_row4_col3, #T_042b5_row4_col4, #T_042b5_row4_col5 {\n",+ " text-align: left;\n",+ " white-space: pre-wrap;\n",+ " word-wrap: break-word;\n",+ " max-width: 400px;\n",+ "}\n",+ "</style>\n",+ "<table id=\"T_042b5\">\n",+ " <thead>\n",+ " <tr>\n",+ " <th class=\"blank level0\" > </th>\n",+ " <th id=\"T_042b5_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",+ " <th id=\"T_042b5_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",+ " <th id=\"T_042b5_level0_col2\" class=\"col_heading level0 col2\" >gold_titles</th>\n",+ " <th id=\"T_042b5_level0_col3\" class=\"col_heading level0 col3\" >output</th>\n",+ " <th id=\"T_042b5_level0_col4\" class=\"col_heading level0 col4\" >tweet_response</th>\n",+ " <th id=\"T_042b5_level0_col5\" class=\"col_heading level0 col5\" >metric</th>\n",+ " </tr>\n",+ " </thead>\n",+ " <tbody>\n",+ " <tr>\n",+ " <th id=\"T_042b5_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",+ " <td id=\"T_042b5_row0_col0\" class=\"data row0 col0\" >Who was a producer who produced albums for both rock bands Juke Karten and Thirty Seconds to Mars?</td>\n",+ " <td id=\"T_042b5_row0_col1\" class=\"data row0 col1\" >Brian Virtue</td>\n",+ " <td id=\"T_042b5_row0_col2\" class=\"data row0 col2\" >{'Levolution (album)', 'Thirty Seconds to Mars'}</td>\n",+ " <td id=\"T_042b5_row0_col3\" class=\"data row0 col3\" >Brian Virtue, who has worked with bands like Jane's Addiction and Velvet Revolver, produced albums for both Juke Kartel and Thirty Seconds to Mars, showcasing...</td>\n",+ " <td id=\"T_042b5_row0_col4\" class=\"data row0 col4\" >Brian Virtue, who has worked with bands like Jane's Addiction and Velvet Revolver, produced albums for both Juke Kartel and Thirty Seconds to Mars, showcasing...</td>\n",+ " <td id=\"T_042b5_row0_col5\" class=\"data row0 col5\" >1.0</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_042b5_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",+ " <td id=\"T_042b5_row1_col0\" class=\"data row1 col0\" >Are both the University of Chicago and Syracuse University public universities? </td>\n",+ " <td id=\"T_042b5_row1_col1\" class=\"data row1 col1\" >no</td>\n",+ " <td id=\"T_042b5_row1_col2\" class=\"data row1 col2\" >{'University of Chicago', 'Syracuse University'}</td>\n",+ " <td id=\"T_042b5_row1_col3\" class=\"data row1 col3\" > No, only Syracuse University is a public university. The University of Chicago is a private research university. #Syracuse #University #Chicago #Public #Private</td>\n",+ " <td id=\"T_042b5_row1_col4\" class=\"data row1 col4\" > No, only Syracuse University is a public university. The University of Chicago is a private research university. #Syracuse #University #Chicago #Public #Private</td>\n",+ " <td id=\"T_042b5_row1_col5\" class=\"data row1 col5\" >0.3333333333333333</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_042b5_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",+ " <td id=\"T_042b5_row2_col0\" class=\"data row2 col0\" >In what region was Eddy Mazzoleni born?</td>\n",+ " <td id=\"T_042b5_row2_col1\" class=\"data row2 col1\" >Lombardy, northern Italy</td>\n",+ " <td id=\"T_042b5_row2_col2\" class=\"data row2 col2\" >{'Bergamo', 'Eddy Mazzoleni'}</td>\n",+ " <td id=\"T_042b5_row2_col3\" class=\"data row2 col3\" > Eddy Mazzoleni, Italian professional cyclist, was born in Bergamo, Italy on July 29, 1973. #cyclist #Italy #Bergamo</td>\n",+ " <td id=\"T_042b5_row2_col4\" class=\"data row2 col4\" > Eddy Mazzoleni, Italian professional cyclist, was born in Bergamo, Italy on July 29, 1973. #cyclist #Italy #Bergamo</td>\n",+ " <td id=\"T_042b5_row2_col5\" class=\"data row2 col5\" >0.0</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_042b5_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",+ " <td id=\"T_042b5_row3_col0\" class=\"data row3 col0\" >Who edited the 1990 American romantic comedy film directed by Garry Marshall?</td>\n",+ " <td id=\"T_042b5_row3_col1\" class=\"data row3 col1\" >Raja Raymond Gosnell</td>\n",+ " <td id=\"T_042b5_row3_col2\" class=\"data row3 col2\" >{'Raja Gosnell', 'Pretty Woman'}</td>\n",+ " <td id=\"T_042b5_row3_col3\" class=\"data row3 col3\" > J. F. Lawton edited the 1990 American romantic comedy film directed by Garry Marshall. #PrettyWoman #GarryMarshall #JFLawton</td>\n",+ " <td id=\"T_042b5_row3_col4\" class=\"data row3 col4\" > J. F. Lawton edited the 1990 American romantic comedy film directed by Garry Marshall. #PrettyWoman #GarryMarshall #JFLawton</td>\n",+ " <td id=\"T_042b5_row3_col5\" class=\"data row3 col5\" >0.0</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_042b5_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",+ " <td id=\"T_042b5_row4_col0\" class=\"data row4 col0\" >Burrs Country Park railway station is what stop on the railway line that runs between Heywood and Rawtenstall</td>\n",+ " <td id=\"T_042b5_row4_col1\" class=\"data row4 col1\" >seventh</td>\n",+ " <td id=\"T_042b5_row4_col2\" class=\"data row4 col2\" >{'East Lancashire Railway', 'Burrs Country Park railway station'}</td>\n",+ " <td id=\"T_042b5_row4_col3\" class=\"data row4 col3\" > Burrs Country Park railway station is the seventh stop on the East Lancashire Railway line that runs between Heywood and Rawtenstall.</td>\n",+ " <td id=\"T_042b5_row4_col4\" class=\"data row4 col4\" > Burrs Country Park railway station is the seventh stop on the East Lancashire Railway line that runs between Heywood and Rawtenstall.</td>\n",+ " <td id=\"T_042b5_row4_col5\" class=\"data row4 col5\" >1.0</td>\n",+ " </tr>\n",+ " </tbody>\n",+ "</table>\n"+ ],+ "text/plain": [+ "<pandas.io.formats.style.Styler at 0x7f08b9f75490>"+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ },+ {+ "data": {+ "text/html": [+ "\n",+ " <div style='\n",+ " text-align: center; \n",+ " font-size: 16px; \n",+ " font-weight: bold; \n",+ " color: #555; \n",+ " margin: 10px 0;'>\n",+ " ... 145 more rows not displayed ...\n",+ " </div>\n",+ " "+ ],+ "text/plain": [+ "<IPython.core.display.HTML object>"+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ },+ {+ "data": {+ "text/plain": [+ "42.67"+ ]+ },+ "execution_count": 8,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "evaluate = Evaluate(metric=metric, devset=devset, num_threads=8, display_progress=True, display_table=5)\n",+ "evaluate(zeroshot_chain)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Okay, cool. Our `zeroshot_chain` gets about **43%** on the 150 questions from the devset.\n",+ "\n",+ "The table above shows some examples. For instance:\n",+ "\n",+ "- **Question**: Who was a producer who produced albums for both rock bands Juke Karten and Thirty Seconds to Mars?\t\n",+ "- **Tweet**: Brian Virtue, who has worked with bands like Jane's Addiction and Velvet Revolver, produced albums for both Juke Kartel and Thirty Seconds to Mars, showcasing... [truncated]\n",+ "- **Metric**: 1.0 (A tweet that is correct, faithful, and engaging!*)\n",+ "\n",+ "footnote: * At least according to our metric, which is just a DSPy program, so _it too_ can be optimized if you'd like! Topic for another notebook, though."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 6) Optimizing the module\n",+ "\n",+ "DSPy has many optimizers, but the de-facto default one currently is `BootstrapFewShotWithRandomSearch`.\n",+ "\n",+ "**If you're curious how it works:** This optimizer works by running your program (in this case, `zeroshot_chain`) on `trainset` questions. Each time it runs, DSPy will remember the input and output of each LM call. These are called traces, and this particular optimizer will keep track of \"good\" traces (i.e., ones that the metric likes). Then, this optimizer will try to find good ways to leverage these traces as automatic few-shot examples. It will try them out, seeking to maximize the average metric on `valset`. There are many ways to self-generate (bootstrap) examples. There are also many ways to optimize their selection (here, with random search). That's why there are several other optimizers in DSPy."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 9,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Going to sample between 1 and 3 traces per predictor.\n",+ "Will attempt to train 3 candidate sets.\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Average Metric: 22.333333333333336 / 50 (44.7): 100%|██████████| 50/50 [00:00<00:00, 56.06it/s]\n",+ "/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/dspy/evaluate/evaluate.py:126: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n",+ " df = df.applymap(truncate_cell)\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Average Metric: 22.333333333333336 / 50 (44.7%)\n",+ "Score: 44.67 for set: [0]\n",+ "New best score: 44.67 for seed -3\n",+ "Scores so far: [44.67]\n",+ "Best score: 44.67\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Average Metric: 22.333333333333336 / 50 (44.7): 100%|██████████| 50/50 [00:00<00:00, 126.78it/s]\n",+ "/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/dspy/evaluate/evaluate.py:126: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n",+ " df = df.applymap(truncate_cell)\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Average Metric: 22.333333333333336 / 50 (44.7%)\n",+ "Score: 44.67 for set: [16]\n",+ "Scores so far: [44.67, 44.67]\n",+ "Best score: 44.67\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ " 2%|▎ | 5/200 [00:00<00:07, 26.22it/s]\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Bootstrapped 3 full traces after 6 examples in round 0.\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Average Metric: 27.000000000000004 / 50 (54.0): 100%|██████████| 50/50 [00:00<00:00, 70.41it/s]\n",+ "/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/dspy/evaluate/evaluate.py:126: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n",+ " df = df.applymap(truncate_cell)\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Average Metric: 27.000000000000004 / 50 (54.0%)\n",+ "Score: 54.0 for set: [16]\n",+ "New best score: 54.0 for seed -1\n",+ "Scores so far: [44.67, 44.67, 54.0]\n",+ "Best score: 54.0\n",+ "Average of max per entry across top 1 scores: 0.54\n",+ "Average of max per entry across top 2 scores: 0.5933333333333334\n",+ "Average of max per entry across top 3 scores: 0.5933333333333334\n",+ "Average of max per entry across top 5 scores: 0.5933333333333334\n",+ "Average of max per entry across top 8 scores: 0.5933333333333334\n",+ "Average of max per entry across top 9999 scores: 0.5933333333333334\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ " 4%|▍ | 9/200 [00:00<00:06, 28.02it/s]\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Bootstrapped 2 full traces after 10 examples in round 0.\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Average Metric: 25.000000000000004 / 50 (50.0): 100%|██████████| 50/50 [00:00<00:00, 76.56it/s]\n",+ "/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/dspy/evaluate/evaluate.py:126: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n",+ " df = df.applymap(truncate_cell)\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Average Metric: 25.000000000000004 / 50 (50.0%)\n",+ "Score: 50.0 for set: [16]\n",+ "Scores so far: [44.67, 44.67, 54.0, 50.0]\n",+ "Best score: 54.0\n",+ "Average of max per entry across top 1 scores: 0.54\n",+ "Average of max per entry across top 2 scores: 0.5933333333333334\n",+ "Average of max per entry across top 3 scores: 0.6066666666666667\n",+ "Average of max per entry across top 5 scores: 0.6066666666666667\n",+ "Average of max per entry across top 8 scores: 0.6066666666666667\n",+ "Average of max per entry across top 9999 scores: 0.6066666666666667\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ " 0%| | 1/200 [00:00<00:06, 28.50it/s]\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Bootstrapped 1 full traces after 2 examples in round 0.\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Average Metric: 25.666666666666664 / 50 (51.3): 100%|██████████| 50/50 [00:00<00:00, 76.77it/s]\n",+ "/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/dspy/evaluate/evaluate.py:126: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n",+ " df = df.applymap(truncate_cell)\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Average Metric: 25.666666666666664 / 50 (51.3%)\n",+ "Score: 51.33 for set: [16]\n",+ "Scores so far: [44.67, 44.67, 54.0, 50.0, 51.33]\n",+ "Best score: 54.0\n",+ "Average of max per entry across top 1 scores: 0.54\n",+ "Average of max per entry across top 2 scores: 0.5800000000000001\n",+ "Average of max per entry across top 3 scores: 0.6133333333333334\n",+ "Average of max per entry across top 5 scores: 0.6266666666666667\n",+ "Average of max per entry across top 8 scores: 0.6266666666666667\n",+ "Average of max per entry across top 9999 scores: 0.6266666666666667\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ " 1%| | 2/200 [00:00<00:07, 27.13it/s]\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Bootstrapped 1 full traces after 3 examples in round 0.\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Average Metric: 26.0 / 50 (52.0): 100%|██████████| 50/50 [00:00<00:00, 73.05it/s] "+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Average Metric: 26.0 / 50 (52.0%)\n",+ "Score: 52.0 for set: [16]\n",+ "Scores so far: [44.67, 44.67, 54.0, 50.0, 51.33, 52.0]\n",+ "Best score: 54.0\n",+ "Average of max per entry across top 1 scores: 0.54\n",+ "Average of max per entry across top 2 scores: 0.5733333333333335\n",+ "Average of max per entry across top 3 scores: 0.6133333333333334\n",+ "Average of max per entry across top 5 scores: 0.64\n",+ "Average of max per entry across top 8 scores: 0.64\n",+ "Average of max per entry across top 9999 scores: 0.64\n",+ "6 candidate programs found.\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "\n",+ "/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/dspy/evaluate/evaluate.py:126: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n",+ " df = df.applymap(truncate_cell)\n"+ ]+ }+ ],+ "source": [+ "# Set up the optimizer. We'll use very minimal hyperparameters for this example.\n",+ "# Just do random search with ~3 attempts, and in each attempt, bootstrap <= 3 traces.\n",+ "optimizer = BootstrapFewShotWithRandomSearch(metric=metric, max_bootstrapped_demos=3, num_candidate_programs=3)\n",+ "\n",+ "# Now use the optimizer to *compile* the chain. This could take 5-10 minutes, unless it's cached.\n",+ "optimized_chain = optimizer.compile(zeroshot_chain, trainset=trainset, valset=valset)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 7) Evaluating the optimized chain\n",+ "\n",+ "Well, how good is this? _Not every optimization run will magically result in improvement on unseen examples!_ So let's check!\n",+ "\n",+ "First let's ask that question from above."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 10,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "' Eddy Mazzoleni was born in Bergamo, a city in the Lombardy region of Italy. #EddyMazzoleni #Italy #Lombardy'"+ ]+ },+ "execution_count": 10,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "question = \"In what region was Eddy Mazzoleni born?\"\n",+ "\n",+ "optimized_chain.invoke({\"question\": question})"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Nice, anecdotally, it appears a bit more precise than the answer with `zeroshot_chain`. But now let's do some proper evals!"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 11,+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Average Metric: 78.66666666666666 / 150 (52.4): 100%|██████████| 150/150 [00:02<00:00, 74.17it/s] "+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Average Metric: 78.66666666666666 / 150 (52.4%)\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "\n",+ "/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/dspy/evaluate/evaluate.py:126: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n",+ " df = df.applymap(truncate_cell)\n"+ ]+ },+ {+ "data": {+ "text/html": [+ "<style type=\"text/css\">\n",+ "#T_c36cc th {\n",+ " text-align: left;\n",+ "}\n",+ "#T_c36cc td {\n",+ " text-align: left;\n",+ "}\n",+ "#T_c36cc_row0_col0, #T_c36cc_row0_col1, #T_c36cc_row0_col2, #T_c36cc_row0_col3, #T_c36cc_row0_col4, #T_c36cc_row0_col5, #T_c36cc_row1_col0, #T_c36cc_row1_col1, #T_c36cc_row1_col2, #T_c36cc_row1_col3, #T_c36cc_row1_col4, #T_c36cc_row1_col5, #T_c36cc_row2_col0, #T_c36cc_row2_col1, #T_c36cc_row2_col2, #T_c36cc_row2_col3, #T_c36cc_row2_col4, #T_c36cc_row2_col5, #T_c36cc_row3_col0, #T_c36cc_row3_col1, #T_c36cc_row3_col2, #T_c36cc_row3_col3, #T_c36cc_row3_col4, #T_c36cc_row3_col5, #T_c36cc_row4_col0, #T_c36cc_row4_col1, #T_c36cc_row4_col2, #T_c36cc_row4_col3, #T_c36cc_row4_col4, #T_c36cc_row4_col5 {\n",+ " text-align: left;\n",+ " white-space: pre-wrap;\n",+ " word-wrap: break-word;\n",+ " max-width: 400px;\n",+ "}\n",+ "</style>\n",+ "<table id=\"T_c36cc\">\n",+ " <thead>\n",+ " <tr>\n",+ " <th class=\"blank level0\" > </th>\n",+ " <th id=\"T_c36cc_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",+ " <th id=\"T_c36cc_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",+ " <th id=\"T_c36cc_level0_col2\" class=\"col_heading level0 col2\" >gold_titles</th>\n",+ " <th id=\"T_c36cc_level0_col3\" class=\"col_heading level0 col3\" >output</th>\n",+ " <th id=\"T_c36cc_level0_col4\" class=\"col_heading level0 col4\" >tweet_response</th>\n",+ " <th id=\"T_c36cc_level0_col5\" class=\"col_heading level0 col5\" >metric</th>\n",+ " </tr>\n",+ " </thead>\n",+ " <tbody>\n",+ " <tr>\n",+ " <th id=\"T_c36cc_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",+ " <td id=\"T_c36cc_row0_col0\" class=\"data row0 col0\" >Who was a producer who produced albums for both rock bands Juke Karten and Thirty Seconds to Mars?</td>\n",+ " <td id=\"T_c36cc_row0_col1\" class=\"data row0 col1\" >Brian Virtue</td>\n",+ " <td id=\"T_c36cc_row0_col2\" class=\"data row0 col2\" >{'Levolution (album)', 'Thirty Seconds to Mars'}</td>\n",+ " <td id=\"T_c36cc_row0_col3\" class=\"data row0 col3\" >Brian Virtue is a producer who has worked with both Juke Kartel and Thirty Seconds to Mars, helping to create their unique sounds. #BrianVirtue #producer...</td>\n",+ " <td id=\"T_c36cc_row0_col4\" class=\"data row0 col4\" >Brian Virtue is a producer who has worked with both Juke Kartel and Thirty Seconds to Mars, helping to create their unique sounds. #BrianVirtue #producer...</td>\n",+ " <td id=\"T_c36cc_row0_col5\" class=\"data row0 col5\" >1.0</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_c36cc_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",+ " <td id=\"T_c36cc_row1_col0\" class=\"data row1 col0\" >Are both the University of Chicago and Syracuse University public universities? </td>\n",+ " <td id=\"T_c36cc_row1_col1\" class=\"data row1 col1\" >no</td>\n",+ " <td id=\"T_c36cc_row1_col2\" class=\"data row1 col2\" >{'University of Chicago', 'Syracuse University'}</td>\n",+ " <td id=\"T_c36cc_row1_col3\" class=\"data row1 col3\" > Yes, both Northeastern Illinois University and Syracuse University are public universities. #publicuniversity #Chicago #Syracuse</td>\n",+ " <td id=\"T_c36cc_row1_col4\" class=\"data row1 col4\" > Yes, both Northeastern Illinois University and Syracuse University are public universities. #publicuniversity #Chicago #Syracuse</td>\n",+ " <td id=\"T_c36cc_row1_col5\" class=\"data row1 col5\" >0.0</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_c36cc_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",+ " <td id=\"T_c36cc_row2_col0\" class=\"data row2 col0\" >In what region was Eddy Mazzoleni born?</td>\n",+ " <td id=\"T_c36cc_row2_col1\" class=\"data row2 col1\" >Lombardy, northern Italy</td>\n",+ " <td id=\"T_c36cc_row2_col2\" class=\"data row2 col2\" >{'Bergamo', 'Eddy Mazzoleni'}</td>\n",+ " <td id=\"T_c36cc_row2_col3\" class=\"data row2 col3\" > Eddy Mazzoleni was born in Bergamo, a city in the Lombardy region of Italy. #EddyMazzoleni #Italy #Lombardy</td>\n",+ " <td id=\"T_c36cc_row2_col4\" class=\"data row2 col4\" > Eddy Mazzoleni was born in Bergamo, a city in the Lombardy region of Italy. #EddyMazzoleni #Italy #Lombardy</td>\n",+ " <td id=\"T_c36cc_row2_col5\" class=\"data row2 col5\" >1.0</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_c36cc_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",+ " <td id=\"T_c36cc_row3_col0\" class=\"data row3 col0\" >Who edited the 1990 American romantic comedy film directed by Garry Marshall?</td>\n",+ " <td id=\"T_c36cc_row3_col1\" class=\"data row3 col1\" >Raja Raymond Gosnell</td>\n",+ " <td id=\"T_c36cc_row3_col2\" class=\"data row3 col2\" >{'Raja Gosnell', 'Pretty Woman'}</td>\n",+ " <td id=\"T_c36cc_row3_col3\" class=\"data row3 col3\" > Garry Marshall directed and edited the 1990 American romantic comedy film \"Pretty Woman\", starring Richard Gere and Julia Roberts. #PrettyWoman #GarryMarshall #RomanticComedy</td>\n",+ " <td id=\"T_c36cc_row3_col4\" class=\"data row3 col4\" > Garry Marshall directed and edited the 1990 American romantic comedy film \"Pretty Woman\", starring Richard Gere and Julia Roberts. #PrettyWoman #GarryMarshall #RomanticComedy</td>\n",+ " <td id=\"T_c36cc_row3_col5\" class=\"data row3 col5\" >0.0</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_c36cc_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",+ " <td id=\"T_c36cc_row4_col0\" class=\"data row4 col0\" >Burrs Country Park railway station is what stop on the railway line that runs between Heywood and Rawtenstall</td>\n",+ " <td id=\"T_c36cc_row4_col1\" class=\"data row4 col1\" >seventh</td>\n",+ " <td id=\"T_c36cc_row4_col2\" class=\"data row4 col2\" >{'East Lancashire Railway', 'Burrs Country Park railway station'}</td>\n",+ " <td id=\"T_c36cc_row4_col3\" class=\"data row4 col3\" > Burrs Country Park railway station is the seventh stop on the East Lancashire Railway line, which runs between Heywood and Rawtenstall. #EastLancashireRailway #BurrsCountryPark #railwaystation</td>\n",+ " <td id=\"T_c36cc_row4_col4\" class=\"data row4 col4\" > Burrs Country Park railway station is the seventh stop on the East Lancashire Railway line, which runs between Heywood and Rawtenstall. #EastLancashireRailway #BurrsCountryPark #railwaystation</td>\n",+ " <td id=\"T_c36cc_row4_col5\" class=\"data row4 col5\" >1.0</td>\n",+ " </tr>\n",+ " </tbody>\n",+ "</table>\n"+ ],+ "text/plain": [+ "<pandas.io.formats.style.Styler at 0x7f08b98dadc0>"+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ },+ {+ "data": {+ "text/html": [+ "\n",+ " <div style='\n",+ " text-align: center; \n",+ " font-size: 16px; \n",+ " font-weight: bold; \n",+ " color: #555; \n",+ " margin: 10px 0;'>\n",+ " ... 145 more rows not displayed ...\n",+ " </div>\n",+ " "+ ],+ "text/plain": [+ "<IPython.core.display.HTML object>"+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ },+ {+ "data": {+ "text/plain": [+ "52.44"+ ]+ },+ "execution_count": 11,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "evaluate(optimized_chain)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "We started with `zeroshot_chain` at **43%** and now we have **52%**. That's a nice **21%** relative improvement. Not bad!"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 8) Inspecting the optimized chain in action"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 12,+ "metadata": {},+ "outputs": [+ {+ "ename": "AttributeError",+ "evalue": "'super' object has no attribute '__getattr__'",+ "output_type": "error",+ "traceback": [+ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",+ "\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)",+ "Cell \u001b[0;32mIn[12], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m prompt, output \u001b[38;5;241m=\u001b[39m \u001b[43mdspy\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msettings\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mlangchain_history\u001b[49m[\u001b[38;5;241m-\u001b[39m\u001b[38;5;241m1\u001b[39m]\n\u001b[1;32m 3\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mPROMPT:\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m'\u001b[39m, prompt)\n\u001b[1;32m 4\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124mOUTPUT:\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m'\u001b[39m, output)\n",+ "File \u001b[0;32m/scr-ssd/okhattab/miniconda3/envs/py39_jan2024_01/lib/python3.9/site-packages/dsp/utils/settings.py:61\u001b[0m, in \u001b[0;36mSettings.__getattr__\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m 58\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m name \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mconfig:\n\u001b[1;32m 59\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mconfig[name]\n\u001b[0;32m---> 61\u001b[0m \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[38;5;21;43m__getattr__\u001b[39;49m(name)\n",+ "\u001b[0;31mAttributeError\u001b[0m: 'super' object has no attribute '__getattr__'"+ ]+ }+ ],+ "source": [+ "prompt, output = dspy.settings.langchain_history[-1]\n",+ "\n",+ "print('PROMPT:\\n\\n', prompt)\n",+ "print('\\n\\nOUTPUT:\\n\\n', output)"+ ]+ }+ ],+ "metadata": {+ "kernelspec": {+ "display_name": "py39_dec_2023",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.9.18"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 2+}diff --git a/examples/tweets/tweet_metric.py b/examples/tweets/tweet_metric.pynew file mode 100644index 000000000..dc4696fd0--- /dev/null+++ b/examples/tweets/tweet_metric.py@@ -0,0 +1,50 @@+import dspy+from dspy.datasets import HotPotQA++# Load the dataset.+dataset = HotPotQA(train_seed=1, train_size=200, eval_seed=2023, dev_size=200, test_size=0, keep_details=True)++# Tell DSPy that the 'question' field is the input. Any other fields are labels and/or metadata.+trainset = [x.without('id', 'type').with_inputs('question') for x in dataset.train]+devset = [x.without('id', 'type').with_inputs('question') for x in dataset.dev]+valset, devset = devset[:50], devset[50:]+++# Define the signature for autoamtic assessments.+class Assess(dspy.Signature):+ """Assess the quality of a tweet along the specified dimension."""++ context = dspy.InputField(desc='ignore if N/A')+ assessed_text = dspy.InputField()+ assessment_question = dspy.InputField()+ assessment_answer = dspy.OutputField(desc="Yes or No")+++gpt4T = dspy.OpenAI(model='gpt-4-1106-preview', max_tokens=1000, model_type='chat')+retrieve = dspy.Retrieve(k=5)+METRIC = None++def metric(gold, pred, trace=None):+ question, answer, tweet = gold.question, gold.answer, pred.output+ context = retrieve(question).passages++ engaging = "Does the assessed text make for a self-contained, engaging tweet?"+ faithful = "Is the assessed text grounded in the context? Say no if it includes significant facts not in the context."+ correct = f"The text above is should answer `{question}`. The gold answer is `{answer}`."+ correct = f"{correct} Does the assessed text above contain the gold answer?"++ with dspy.context(lm=gpt4T):+ faithful = dspy.Predict(Assess)(context=context, assessed_text=tweet, assessment_question=faithful)+ correct = dspy.Predict(Assess)(context='N/A', assessed_text=tweet, assessment_question=correct)+ engaging = dspy.Predict(Assess)(context='N/A', assessed_text=tweet, assessment_question=engaging)++ correct, engaging, faithful = [m.assessment_answer.split()[0].lower() == 'yes' for m in [correct, engaging, faithful]]+ score = (correct + engaging + faithful) if correct and (len(tweet) <= 280) else 0++ if METRIC is not None:+ if METRIC == 'correct': return correct+ if METRIC == 'engaging': return engaging+ if METRIC == 'faithful': return faithful++ if trace is not None: return score >= 3+ return score / 3.0