Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Update finetune demo

dilara5a895cc5527c
diff --git a/examples/finetuning/basicmh_3_3_6.json b/examples/finetuning/basicmh_3_3_6.json
deleted file mode 100644
index 37be3f494..000000000
--- a/examples/finetuning/basicmh_3_3_6.json
+++ /dev/null
@@ -1,340 +0,0 @@
-{
- "retrieve": {
- "k": 3
- },
- "generate_query[0]": {
- "lm": null,
- "traces": [],
- "train": [],
- "demos": [
- {
- "augmented": true,
- "context": [],
- "question": "Singer Annette Snell died in a airplane crash during a forced landing on a highway in what state?",
- "rationale": "determine the location of the airplane crash involving singer Annette Snell. First, we need to identify any news articles, reports, or obituaries that mention the details of Annette Snell's death, including the state where the crash occurred.",
- "search_query": "\"Annette Snell airplane crash state\""
- },
- {
- "question": "Which movie is based on a book, Darby O'Gill and the Little People and College Road Trip?",
- "answer": "Darby O'Gill and the Little People",
- "dspy_uuid": "346d7c7a-9f8d-485b-b9a0-cd3e813e4f79",
- "dspy_split": "train"
- },
- {
- "question": "Which company distributed this 1977 American animated film produced by Walt Disney Productions for which Sherman Brothers wrote songs?",
- "answer": "Buena Vista Distribution",
- "dspy_uuid": "1d429a20-dab2-45c8-9c7e-f09c02ff64b7",
- "dspy_split": "train"
- },
- {
- "question": "Which director was nominated for more Academy Awards, Lee Tit or Sidney Lumet?",
- "answer": "14 of his films were nominated",
- "dspy_uuid": "2c8820fb-8a3e-4285-a8f1-f6421ff23bff",
- "dspy_split": "train"
- },
- {
- "question": "A police detective, author and former child actor who appeared on a show that had segments about making food and taking things apart, joined the police department in what year?",
- "answer": "1996",
- "dspy_uuid": "1694f0bc-c3e7-4c3e-8ba6-e7d7afe1c060",
- "dspy_split": "train"
- },
- {
- "question": "Which American actress who made their film debut in the 1995 teen drama \"Kids\" was the co-founder of Voto Latino?",
- "answer": "Rosario Dawson",
- "dspy_uuid": "7aa6f285-aa9f-4302-a740-1ebf9fe25ecd",
- "dspy_split": "train"
- },
- {
- "question": "Who created the TV series which had the storyline Sharongate ?",
- "answer": "Julia Smith and Tony Holland",
- "dspy_uuid": "fa0f165b-2338-4048-b82e-25185e1d5bce",
- "dspy_split": "train"
- },
- {
- "question": "which American actor was Candace Kita guest starred with ",
- "answer": "Bill Murray",
- "dspy_uuid": "6872de0d-e60f-4568-b67b-3c12efcce4df",
- "dspy_split": "train"
- },
- {
- "question": "Remember Me Ballin' is a CD single by Indo G that features an American rapper born in what year?",
- "answer": "1979",
- "dspy_uuid": "c942e7bd-acc4-4f4a-bc52-aec8b6e49567",
- "dspy_split": "train"
- },
- {
- "question": "Both Jack Hill and Bernard McEveety are involved in what occupation?",
- "answer": "film director",
- "dspy_uuid": "0b3e77cf-0913-4a14-a869-80baac72814b",
- "dspy_split": "train"
- },
- {
- "question": "\"The Day I Met Marie\" is a song written by an English songwriter best known as the lead guitarist for what band?",
- "answer": "the Shadows",
- "dspy_uuid": "01cf72ea-4673-4b75-99d1-4ccddd05a74a",
- "dspy_split": "train"
- },
- {
- "question": "Are both Ralph Saenz and Roddy Woomble credited as writers?",
- "answer": "no",
- "dspy_uuid": "9c678b8b-7e9a-4ac2-99fb-705af4bcabd2",
- "dspy_split": "train"
- },
- {
- "question": "The Montenegrin Mountain Hound and Mudhol Hound are both used for what common purpose?",
- "answer": "hunting",
- "dspy_uuid": "b5e89309-95e2-4e15-b1a4-04653e0b79c1",
- "dspy_split": "train"
- },
- {
- "question": "Carlo Ignazio Monza's music formed a ballet based on what 18th century play? ",
- "answer": "Commedia dell'arte",
- "dspy_uuid": "5237b998-157b-4be0-8911-c57aac61113e",
- "dspy_split": "train"
- },
- {
- "question": "Who lived longer, Al-Ghazali or Averroes (Ibn Rushd)?",
- "answer": "Ibn Rushd",
- "dspy_uuid": "5198340d-31cb-468e-b4dd-2a9ffdba341d",
- "dspy_split": "train"
- },
- {
- "question": "Which magazine has published articles by Scott Shaw, Tae Kwon Do Times or Southwest Art?",
- "answer": "Tae Kwon Do Times",
- "dspy_uuid": "077a1105-3c21-4d34-8cb7-bab9f6f1076c",
- "dspy_split": "train"
- }
- ],
- "signature_instructions": "Given the fields `context`, `question`, produce the fields `search_query`.",
- "signature_prefix": "Search Query:",
- "extended_signature_instructions": "Given the fields `context`, `question`, produce the fields `search_query`.",
- "extended_signature_prefix": "Search Query:"
- },
- "generate_query[1]": {
- "lm": null,
- "traces": [],
- "train": [],
- "demos": [
- {
- "augmented": true,
- "context": [
- "Annette Snell | Annette Snell (March 22, 1945 \u2013 April 4, 1977) was an American rhythm and blues singer who recorded in the 1960s and 1970s. She died in the Southern Airways Flight 242 crash.",
- "Annette Herfkens | Annette Herfkens (Maracaibo, 29 April 1961) is the sole survivor of Vietnam Airlines Flight 474 which crashed into a mountain on November 14, 1992 in Vietnam. She survived eight days with multiple injuries and sustained herself only on rainwater. The plane, a Soviet-made Yakovlev Yak-40 jet built in 1976, was carrying 25 passengers and six crew when it crashed en route to Nha Trang Airport. Some passengers survived the initial impact but died before they could be rescued. Herfkens' fianc\u00e9, who was travelling with her, died instantly upon impact.",
- "1947 Oregon Beechcraft Bonanza crash | On October 28, 1947, Oregon Governor Earl Snell, second in line of succession Oregon State Senate President Marshall Cornett, and Oregon Secretary of State Robert S. Farrell, Jr. were killed in a crash of a Beechcraft Bonanza in stormy weather southwest of Dog Lake in rural Lake County, Oregon. The trio died along with the aircraft's pilot, who was taking them to an isolated area to hunt Canada geese. The tragedy propelled Oregon Speaker of the House John Hubert Hall to the Governorship of Oregon."
- ],
- "question": "Singer Annette Snell died in a airplane crash during a forced landing on a highway in what state?",
- "rationale": "produce the search_query. We need to find information on the circumstances surrounding Annette Snell's death in the airplane crash. Since the context mentions that Annette Snell died in the Southern Airways Flight 242 crash, we should look up details specifically about this crash, including where it occurred. The question asks about the state of the forced landing, so we will search for information regarding the location of this incident.",
- "search_query": "\"Southern Airways Flight 242 crash location state\""
- },
- {
- "question": "Which American actress who made their film debut in the 1995 teen drama \"Kids\" was the co-founder of Voto Latino?",
- "answer": "Rosario Dawson",
- "dspy_uuid": "7aa6f285-aa9f-4302-a740-1ebf9fe25ecd",
- "dspy_split": "train"
- },
- {
- "question": "Which director was nominated for more Academy Awards, Lee Tit or Sidney Lumet?",
- "answer": "14 of his films were nominated",
- "dspy_uuid": "2c8820fb-8a3e-4285-a8f1-f6421ff23bff",
- "dspy_split": "train"
- },
- {
- "question": "Carlo Ignazio Monza's music formed a ballet based on what 18th century play? ",
- "answer": "Commedia dell'arte",
- "dspy_uuid": "5237b998-157b-4be0-8911-c57aac61113e",
- "dspy_split": "train"
- },
- {
- "question": "Which company distributed this 1977 American animated film produced by Walt Disney Productions for which Sherman Brothers wrote songs?",
- "answer": "Buena Vista Distribution",
- "dspy_uuid": "1d429a20-dab2-45c8-9c7e-f09c02ff64b7",
- "dspy_split": "train"
- },
- {
- "question": "\"The Day I Met Marie\" is a song written by an English songwriter best known as the lead guitarist for what band?",
- "answer": "the Shadows",
- "dspy_uuid": "01cf72ea-4673-4b75-99d1-4ccddd05a74a",
- "dspy_split": "train"
- },
- {
- "question": "Which magazine has published articles by Scott Shaw, Tae Kwon Do Times or Southwest Art?",
- "answer": "Tae Kwon Do Times",
- "dspy_uuid": "077a1105-3c21-4d34-8cb7-bab9f6f1076c",
- "dspy_split": "train"
- },
- {
- "question": "Both Jack Hill and Bernard McEveety are involved in what occupation?",
- "answer": "film director",
- "dspy_uuid": "0b3e77cf-0913-4a14-a869-80baac72814b",
- "dspy_split": "train"
- },
- {
- "question": "Who lived longer, Al-Ghazali or Averroes (Ibn Rushd)?",
- "answer": "Ibn Rushd",
- "dspy_uuid": "5198340d-31cb-468e-b4dd-2a9ffdba341d",
- "dspy_split": "train"
- },
- {
- "question": "Remember Me Ballin' is a CD single by Indo G that features an American rapper born in what year?",
- "answer": "1979",
- "dspy_uuid": "c942e7bd-acc4-4f4a-bc52-aec8b6e49567",
- "dspy_split": "train"
- },
- {
- "question": "Which movie is based on a book, Darby O'Gill and the Little People and College Road Trip?",
- "answer": "Darby O'Gill and the Little People",
- "dspy_uuid": "346d7c7a-9f8d-485b-b9a0-cd3e813e4f79",
- "dspy_split": "train"
- },
- {
- "question": "Who created the TV series which had the storyline Sharongate ?",
- "answer": "Julia Smith and Tony Holland",
- "dspy_uuid": "fa0f165b-2338-4048-b82e-25185e1d5bce",
- "dspy_split": "train"
- },
- {
- "question": "which American actor was Candace Kita guest starred with ",
- "answer": "Bill Murray",
- "dspy_uuid": "6872de0d-e60f-4568-b67b-3c12efcce4df",
- "dspy_split": "train"
- },
- {
- "question": "The Montenegrin Mountain Hound and Mudhol Hound are both used for what common purpose?",
- "answer": "hunting",
- "dspy_uuid": "b5e89309-95e2-4e15-b1a4-04653e0b79c1",
- "dspy_split": "train"
- },
- {
- "question": "Are both Ralph Saenz and Roddy Woomble credited as writers?",
- "answer": "no",
- "dspy_uuid": "9c678b8b-7e9a-4ac2-99fb-705af4bcabd2",
- "dspy_split": "train"
- },
- {
- "question": "A police detective, author and former child actor who appeared on a show that had segments about making food and taking things apart, joined the police department in what year?",
- "answer": "1996",
- "dspy_uuid": "1694f0bc-c3e7-4c3e-8ba6-e7d7afe1c060",
- "dspy_split": "train"
- }
- ],
- "signature_instructions": "Given the fields `context`, `question`, produce the fields `search_query`.",
- "signature_prefix": "Search Query:",
- "extended_signature_instructions": "Given the fields `context`, `question`, produce the fields `search_query`.",
- "extended_signature_prefix": "Search Query:"
- },
- "generate_answer": {
- "lm": null,
- "traces": [],
- "train": [],
- "demos": [
- {
- "augmented": true,
- "context": [
- "Annette Snell | Annette Snell (March 22, 1945 \u2013 April 4, 1977) was an American rhythm and blues singer who recorded in the 1960s and 1970s. She died in the Southern Airways Flight 242 crash.",
- "Annette Herfkens | Annette Herfkens (Maracaibo, 29 April 1961) is the sole survivor of Vietnam Airlines Flight 474 which crashed into a mountain on November 14, 1992 in Vietnam. She survived eight days with multiple injuries and sustained herself only on rainwater. The plane, a Soviet-made Yakovlev Yak-40 jet built in 1976, was carrying 25 passengers and six crew when it crashed en route to Nha Trang Airport. Some passengers survived the initial impact but died before they could be rescued. Herfkens' fianc\u00e9, who was travelling with her, died instantly upon impact.",
- "1947 Oregon Beechcraft Bonanza crash | On October 28, 1947, Oregon Governor Earl Snell, second in line of succession Oregon State Senate President Marshall Cornett, and Oregon Secretary of State Robert S. Farrell, Jr. were killed in a crash of a Beechcraft Bonanza in stormy weather southwest of Dog Lake in rural Lake County, Oregon. The trio died along with the aircraft's pilot, who was taking them to an isolated area to hunt Canada geese. The tragedy propelled Oregon Speaker of the House John Hubert Hall to the Governorship of Oregon.",
- "Southern Airways Flight 242 | Southern Airways Flight 242 was a DC-9-31 jet, registered N1335U, that executed a forced landing on a highway in New Hope, Paulding County, Georgia, United States after suffering hail damage and losing thrust on both engines in a severe thunderstorm on April 4, 1977.",
- "Southern Airways Flight 932 | Southern Airways Flight 932 was a chartered Southern Airways Douglas DC-9 domestic United States commercial jet flight from Stallings Field (ISO) in Kinston, North Carolina, to Huntington Tri-State Airport\/Milton J. Ferguson Field (HTS) in Ceredo, West Virginia. At 7:36 pm on November 14, 1970, the aircraft crashed into a hill just short of the Tri-State Airport, killing all 75 people on board.",
- "Southern Airways Flight 49 | The hijacking of Southern Airways Flight 49 started on November 10, 1972 in Birmingham, Alabama, stretching over 30 hours and 4000 mi , not ending until the next evening in Havana, Cuba. Melvin Cale, Louis Moore, and Henry D. Jackson, Jr. successfully hijacked a Southern Airways Douglas DC-9 that was scheduled to fly from Memphis, Tennessee to Miami, Florida via Birmingham and Montgomery, Alabama and Orlando, Florida. The three were each facing criminal charges for unrelated incidents. 34 people, including 31 passengers and 3 crew members, were aboard the airplane when it was hijacked. The hijackers' threat to crash the aircraft into a nuclear reactor led directly to the requirement that U.S. airline passengers be physically screened, beginning January 5, 1973."
- ],
- "question": "Singer Annette Snell died in a airplane crash during a forced landing on a highway in what state?",
- "rationale": "determine the state where the forced landing occurred. The context mentions Southern Airways Flight 242, which executed a forced landing in New Hope, Paulding County, Georgia, United States. Since Annette Snell died in this crash and it was in Georgia, we can conclude that the state is Georgia.",
- "answer": "Georgia"
- },
- {
- "question": "Both Jack Hill and Bernard McEveety are involved in what occupation?",
- "answer": "film director",
- "dspy_uuid": "0b3e77cf-0913-4a14-a869-80baac72814b",
- "dspy_split": "train"
- },
- {
- "question": "Which magazine has published articles by Scott Shaw, Tae Kwon Do Times or Southwest Art?",
- "answer": "Tae Kwon Do Times",
- "dspy_uuid": "077a1105-3c21-4d34-8cb7-bab9f6f1076c",
- "dspy_split": "train"
- },
- {
- "question": "Which movie is based on a book, Darby O'Gill and the Little People and College Road Trip?",
- "answer": "Darby O'Gill and the Little People",
- "dspy_uuid": "346d7c7a-9f8d-485b-b9a0-cd3e813e4f79",
- "dspy_split": "train"
- },
- {
- "question": "Who created the TV series which had the storyline Sharongate ?",
- "answer": "Julia Smith and Tony Holland",
- "dspy_uuid": "fa0f165b-2338-4048-b82e-25185e1d5bce",
- "dspy_split": "train"
- },
- {
- "question": "Which company distributed this 1977 American animated film produced by Walt Disney Productions for which Sherman Brothers wrote songs?",
- "answer": "Buena Vista Distribution",
- "dspy_uuid": "1d429a20-dab2-45c8-9c7e-f09c02ff64b7",
- "dspy_split": "train"
- },
- {
- "question": "Remember Me Ballin' is a CD single by Indo G that features an American rapper born in what year?",
- "answer": "1979",
- "dspy_uuid": "c942e7bd-acc4-4f4a-bc52-aec8b6e49567",
- "dspy_split": "train"
- },
- {
- "question": "Who lived longer, Al-Ghazali or Averroes (Ibn Rushd)?",
- "answer": "Ibn Rushd",
- "dspy_uuid": "5198340d-31cb-468e-b4dd-2a9ffdba341d",
- "dspy_split": "train"
- },
- {
- "question": "The Montenegrin Mountain Hound and Mudhol Hound are both used for what common purpose?",
- "answer": "hunting",
- "dspy_uuid": "b5e89309-95e2-4e15-b1a4-04653e0b79c1",
- "dspy_split": "train"
- },
- {
- "question": "A police detective, author and former child actor who appeared on a show that had segments about making food and taking things apart, joined the police department in what year?",
- "answer": "1996",
- "dspy_uuid": "1694f0bc-c3e7-4c3e-8ba6-e7d7afe1c060",
- "dspy_split": "train"
- },
- {
- "question": "\"The Day I Met Marie\" is a song written by an English songwriter best known as the lead guitarist for what band?",
- "answer": "the Shadows",
- "dspy_uuid": "01cf72ea-4673-4b75-99d1-4ccddd05a74a",
- "dspy_split": "train"
- },
- {
- "question": "Carlo Ignazio Monza's music formed a ballet based on what 18th century play? ",
- "answer": "Commedia dell'arte",
- "dspy_uuid": "5237b998-157b-4be0-8911-c57aac61113e",
- "dspy_split": "train"
- },
- {
- "question": "Which American actress who made their film debut in the 1995 teen drama \"Kids\" was the co-founder of Voto Latino?",
- "answer": "Rosario Dawson",
- "dspy_uuid": "7aa6f285-aa9f-4302-a740-1ebf9fe25ecd",
- "dspy_split": "train"
- },
- {
- "question": "Are both Ralph Saenz and Roddy Woomble credited as writers?",
- "answer": "no",
- "dspy_uuid": "9c678b8b-7e9a-4ac2-99fb-705af4bcabd2",
- "dspy_split": "train"
- },
- {
- "question": "which American actor was Candace Kita guest starred with ",
- "answer": "Bill Murray",
- "dspy_uuid": "6872de0d-e60f-4568-b67b-3c12efcce4df",
- "dspy_split": "train"
- },
- {
- "question": "Which director was nominated for more Academy Awards, Lee Tit or Sidney Lumet?",
- "answer": "14 of his films were nominated",
- "dspy_uuid": "2c8820fb-8a3e-4285-a8f1-f6421ff23bff",
- "dspy_split": "train"
- }
- ],
- "signature_instructions": "Given the fields `context`, `question`, produce the fields `answer`.",
- "signature_prefix": "Answer:",
- "extended_signature_instructions": "Given the fields `context`, `question`, produce the fields `answer`.",
- "extended_signature_prefix": "Answer:"
- }
-}
\ No newline at end of file
diff --git a/examples/finetuning/finetuning_demo.ipynb b/examples/finetuning/finetuning_demo.ipynb
new file mode 100644
index 000000000..13d764abf
--- /dev/null
+++ b/examples/finetuning/finetuning_demo.ipynb
@@ -0,0 +1,1188 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "<img src=\"../../docs/images/DSPy8.png\" alt=\"DSPy7 Image\" height=\"150\"/>\n",
+ "\n",
+ "# DSPy Finetuning Demo\n",
+ "\n",
+ "[<img align=\"center\" src=\"https://colab.research.google.com/assets/colab-badge.svg\" />](https://colab.research.google.com/github/stanfordnlp/dspy/blob/main/examples/finetuning/finetuning_demo.ipynb)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "This is a demo notebook showing an example of how fine-tuning models work in DSPy.\n",
+ "In particular, we fine-tune `OpenAI`s `GPT-4o-mini` on `HotPotQA`.\n",
+ "Note that the fine-tuning features in DSPy are experimental and are subject to change."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## I. Module Setup"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The fine-tuning features are available on a special branch named `demo_finetune`.\n",
+ "The following code block checks out the DSPy repository, switches to this branch, and installs DSPy locally."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Replace the following with the path to the DSPy repository on your machine,\n",
+ "# if you have already cloned it. Make sure your working tree is clean! If you\n",
+ "# don't specify a path, we will clone the repository to the current directory.\n",
+ "# \n",
+ "# Leave as is to run on Google Colab.\n",
+ "repo_path = ''\n",
+ "\n",
+ "if not repo_path:\n",
+ " try:\n",
+ " repo_path = 'dspy'\n",
+ " !git clone https://github.com/stanfordnlp/dspy $repo_path\n",
+ " except:\n",
+ " raise Exception(\"Could not clone and install the repository!\")\n",
+ "\n",
+ "\n",
+ "!cd $repo_path && git checkout demo_finetune # Switch to the branch\n",
+ "!pip install -e $repo_path # Install the local package"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We are making use of a demo cache for this notebook, located in \"dspy/examples/finetuning/demo_cache\" in the `demo_finetuning` branch of the DSPy repository.\n",
+ "This allows us to utilize cached calls to the OpenAI API.\n",
+ "If you are not modifying this notebook, you don't need to set up the `OPENAI_API_KEY`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "\n",
+ "demo_cache_path = os.path.join(repo_path, \"dspy\", \"examples\", \"finetuning\", \"demo_cache\")\n",
+ "os.environ[\"DSP_CACHEDIR\"] = demo_cache_path"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If you would like to modify the notebook and change the requests being made to the OpenAI API, you should set the `OPENAI_API_KEY`.\n",
+ "The next block demonstrates how."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Here are the different options for setting the OPENAI_API_KEY.\n",
+ "# (1) If you are running this notebook locally, you can set it in your shell:\n",
+ "# export OPENAI_API_KEY=<YOUR_OPENAI_API_KEY>\n",
+ "\n",
+ "# (2) If you are running this notebook in a Google Colab, you can set it in\n",
+ "# the secrets panel and import by uncommenting the following lines:\n",
+ "# from google.colab import userdata\n",
+ "# os.environ[\"OPENAI_API_KEY\"] = userdata.get(\"OPENAI_API_KEY\")\n",
+ "\n",
+ "# (3) You can set it directly in the notebook by uncommenting the following\n",
+ "# line with the appropriate key. If you pick this option, make sure to\n",
+ "# remove your key before sharing your notebook or committing to a\n",
+ "# repository.\n",
+ "# os.environ[\"OPENAI_API_KEY\"] = <OPENAI_API_KEY>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We will make use of a random number generator in this notebook.\n",
+ "We are creating a `Random` object here to ensure that our notebook is reproducible."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import random\n",
+ "\n",
+ "rng = random.Random()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## II. Task Setup"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In a typical DSPy fashion, we will start by setting up our task, dataset, metric, models and optimizer settings used for `HotPotQA`!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import dspy\n",
+ "from dspy.datasets import HotPotQA\n",
+ "from dspy.evaluate import Evaluate\n",
+ "from dsp.utils.utils import deduplicate\n",
+ "\n",
+ "\n",
+ "# We are setting the experimental flag to True to make use of the fine-tuning\n",
+ "# features that are still in development.\n",
+ "dspy.settings.configure(experimental=True)\n",
+ "\n",
+ "# Define the program\n",
+ "class BasicMH(dspy.Module):\n",
+ " def __init__(self, passages_per_hop=3, num_hops=2):\n",
+ " super().__init__()\n",
+ " self.num_hops = 2\n",
+ " self.retrieve = dspy.Retrieve(k=passages_per_hop)\n",
+ " self.generate_query = [dspy.ChainOfThought(\"context, question -> search_query\") for _ in range(self.num_hops)]\n",
+ " self.generate_answer = dspy.ChainOfThought(\"context, question -> answer\")\n",
+ " \n",
+ " def forward(self, question):\n",
+ " context = []\n",
+ " \n",
+ " for hop in range(self.num_hops):\n",
+ " search_query = self.generate_query[hop](context=context, question=question).search_query\n",
+ " passages = self.retrieve(search_query).passages\n",
+ " context = deduplicate(context + passages)\n",
+ "\n",
+ " answer = self.generate_answer(context=context, question=question).copy(context=context)\n",
+ " return answer\n",
+ "\n",
+ "# Prepare the dataset\n",
+ "TRAIN_SIZE = 1000\n",
+ "DEV_SIZE = 500\n",
+ "dataset = HotPotQA(train_seed=1, eval_seed=2023, test_size=0, only_hard_examples=True)\n",
+ "trainset = [x.with_inputs('question') for x in dataset.train][:TRAIN_SIZE]\n",
+ "devset = [x.with_inputs('question') for x in dataset.dev][:DEV_SIZE]\n",
+ "\n",
+ "# Prepare the metric and evaluator\n",
+ "NUM_THREADS = 12\n",
+ "metric = dspy.evaluate.answer_exact_match\n",
+ "evaluate = Evaluate(devset=devset, metric=metric, num_threads=NUM_THREADS, display_progress=True)\n",
+ "\n",
+ "# Prepare the retriever model\n",
+ "COLBERT_V2_ENDPOINT = \"http://20.102.90.50:2017/wiki17_abstracts\"\n",
+ "retriever = dspy.ColBERTv2(url=COLBERT_V2_ENDPOINT)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Here we specifiy the parameters required for `BootstrapFewShotWithRandomSearch` at the beginning of our notebook so that they can be re-used."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from dspy.teleprompt.random_search import BootstrapFewShotWithRandomSearch\n",
+ "\n",
+ "# Define the hyperparameters for prompt optimization\n",
+ "MAX_BOOTSTRAPPED_DEMOS = 3\n",
+ "MAX_LABELED_DEMOS = 3\n",
+ "NUM_CANDIDATE_PROGRAMS = 6\n",
+ "OPTIMIZER_NUM_TRAIN = 100\n",
+ "OPTIMIZER_NUM_VAL = 150"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In the next block, we are setting up our fine-tunable LM. In `DSPy`, all the\n",
+ "finetunable LMs are subclasses of the `TrainableLM` class, which itself is a\n",
+ "subclass of the `LM` class.\n",
+ "This allows us to communicate with `TrainableLM`s using a common interface.\n",
+ "\n",
+ "`TrainableLM`s can be used as regular `LM`s, but they can also be used to fine-tune models!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 5,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Model parameters\n",
+ "GPT_4O_MINI = \"gpt-4o-mini-2024-07-18\"\n",
+ "MAX_TOKENS = 1024\n",
+ "MODEL_PARAMETERS = {\n",
+ " \"max_tokens\": MAX_TOKENS,\n",
+ " \"temperature\": 0,\n",
+ "}\n",
+ "\n",
+ "# Prepare the language model\n",
+ "lm = dspy.TrainableOpenAI(model=GPT_4O_MINI, **MODEL_PARAMETERS)\n",
+ "\n",
+ "# Showing that the model is an instance of the TrainableLM class\n",
+ "isinstance(lm, dspy.TrainableLM)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's evaluate a vanilla program on our LM!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Evaluating the vanilla program on the devset...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 214 / 500 (42.8): 100%|██████████| 500/500 [00:04<00:00, 118.23it/s]\n"
+ ]
+ }
+ ],
+ "source": [
+ "vanilla_program = BasicMH()\n",
+ "\n",
+ "with dspy.context(lm=lm, rm=retriever):\n",
+ " print(\"Evaluating the vanilla program on the devset...\")\n",
+ " vanilla_base_eval = evaluate(vanilla_program)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## III. Prompt Optimize the Base Model"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We start by prompt optimizing the base model, which will allow us to get higher quality bootstraps for training!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Going to sample between 1 and 3 traces per predictor.\n",
+ "Will attempt to bootstrap 6 candidate sets.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 69 / 150 (46.0): 100%|██████████| 150/150 [00:01<00:00, 125.64it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Score: 46.0 for set: [0, 0, 0]\n",
+ "New best score: 46.0 for seed -3\n",
+ "Scores so far: [46.0]\n",
+ "Best score: 46.0\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 77 / 150 (51.3): 100%|██████████| 150/150 [00:00<00:00, 285.04it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Score: 51.33 for set: [3, 3, 3]\n",
+ "New best score: 51.33 for seed -2\n",
+ "Scores so far: [46.0, 51.33]\n",
+ "Best score: 51.33\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ " 5%|▌ | 5/100 [00:00<00:01, 87.84it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Bootstrapped 3 full traces after 6 examples in round 0.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 83 / 150 (55.3): 100%|██████████| 150/150 [00:01<00:00, 118.81it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Score: 55.33 for set: [3, 3, 3]\n",
+ "New best score: 55.33 for seed -1\n",
+ "Scores so far: [46.0, 51.33, 55.33]\n",
+ "Best score: 55.33\n",
+ "Average of max per entry across top 1 scores: 0.5533333333333333\n",
+ "Average of max per entry across top 2 scores: 0.6066666666666667\n",
+ "Average of max per entry across top 3 scores: 0.6266666666666667\n",
+ "Average of max per entry across top 5 scores: 0.6266666666666667\n",
+ "Average of max per entry across top 8 scores: 0.6266666666666667\n",
+ "Average of max per entry across top 9999 scores: 0.6266666666666667\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ " 6%|▌ | 6/100 [00:00<00:01, 90.50it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Bootstrapped 2 full traces after 7 examples in round 0.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 80 / 150 (53.3): 100%|██████████| 150/150 [00:01<00:00, 115.86it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Score: 53.33 for set: [3, 3, 3]\n",
+ "Scores so far: [46.0, 51.33, 55.33, 53.33]\n",
+ "Best score: 55.33\n",
+ "Average of max per entry across top 1 scores: 0.5533333333333333\n",
+ "Average of max per entry across top 2 scores: 0.6\n",
+ "Average of max per entry across top 3 scores: 0.6333333333333333\n",
+ "Average of max per entry across top 5 scores: 0.6466666666666666\n",
+ "Average of max per entry across top 8 scores: 0.6466666666666666\n",
+ "Average of max per entry across top 9999 scores: 0.6466666666666666\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ " 2%|▏ | 2/100 [00:00<00:01, 87.84it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Bootstrapped 1 full traces after 3 examples in round 0.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 79 / 150 (52.7): 100%|██████████| 150/150 [00:01<00:00, 109.82it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Score: 52.67 for set: [3, 3, 3]\n",
+ "Scores so far: [46.0, 51.33, 55.33, 53.33, 52.67]\n",
+ "Best score: 55.33\n",
+ "Average of max per entry across top 1 scores: 0.5533333333333333\n",
+ "Average of max per entry across top 2 scores: 0.6\n",
+ "Average of max per entry across top 3 scores: 0.6133333333333333\n",
+ "Average of max per entry across top 5 scores: 0.6466666666666666\n",
+ "Average of max per entry across top 8 scores: 0.6466666666666666\n",
+ "Average of max per entry across top 9999 scores: 0.6466666666666666\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ " 2%|▏ | 2/100 [00:00<00:00, 124.16it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Bootstrapped 1 full traces after 3 examples in round 0.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 85 / 150 (56.7): 100%|██████████| 150/150 [00:01<00:00, 136.13it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Score: 56.67 for set: [3, 3, 3]\n",
+ "New best score: 56.67 for seed 2\n",
+ "Scores so far: [46.0, 51.33, 55.33, 53.33, 52.67, 56.67]\n",
+ "Best score: 56.67\n",
+ "Average of max per entry across top 1 scores: 0.5666666666666667\n",
+ "Average of max per entry across top 2 scores: 0.6133333333333333\n",
+ "Average of max per entry across top 3 scores: 0.6333333333333333\n",
+ "Average of max per entry across top 5 scores: 0.64\n",
+ "Average of max per entry across top 8 scores: 0.6533333333333333\n",
+ "Average of max per entry across top 9999 scores: 0.6533333333333333\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ " 1%| | 1/100 [00:00<00:01, 75.49it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Bootstrapped 1 full traces after 2 examples in round 0.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 81 / 150 (54.0): 100%|██████████| 150/150 [00:01<00:00, 130.89it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Score: 54.0 for set: [3, 3, 3]\n",
+ "Scores so far: [46.0, 51.33, 55.33, 53.33, 52.67, 56.67, 54.0]\n",
+ "Best score: 56.67\n",
+ "Average of max per entry across top 1 scores: 0.5666666666666667\n",
+ "Average of max per entry across top 2 scores: 0.6133333333333333\n",
+ "Average of max per entry across top 3 scores: 0.6466666666666666\n",
+ "Average of max per entry across top 5 scores: 0.66\n",
+ "Average of max per entry across top 8 scores: 0.66\n",
+ "Average of max per entry across top 9999 scores: 0.66\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ " 1%| | 1/100 [00:00<00:01, 75.95it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Bootstrapped 1 full traces after 2 examples in round 0.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 75 / 150 (50.0): 100%|██████████| 150/150 [00:01<00:00, 135.02it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Score: 50.0 for set: [3, 3, 3]\n",
+ "Scores so far: [46.0, 51.33, 55.33, 53.33, 52.67, 56.67, 54.0, 50.0]\n",
+ "Best score: 56.67\n",
+ "Average of max per entry across top 1 scores: 0.5666666666666667\n",
+ "Average of max per entry across top 2 scores: 0.6133333333333333\n",
+ "Average of max per entry across top 3 scores: 0.6466666666666666\n",
+ "Average of max per entry across top 5 scores: 0.66\n",
+ "Average of max per entry across top 8 scores: 0.66\n",
+ "Average of max per entry across top 9999 scores: 0.66\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ " 3%|▎ | 3/100 [00:00<00:00, 118.88it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Bootstrapped 3 full traces after 4 examples in round 0.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 79 / 150 (52.7): 100%|██████████| 150/150 [00:01<00:00, 130.65it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Score: 52.67 for set: [3, 3, 3]\n",
+ "Scores so far: [46.0, 51.33, 55.33, 53.33, 52.67, 56.67, 54.0, 50.0, 52.67]\n",
+ "Best score: 56.67\n",
+ "Average of max per entry across top 1 scores: 0.5666666666666667\n",
+ "Average of max per entry across top 2 scores: 0.6133333333333333\n",
+ "Average of max per entry across top 3 scores: 0.6466666666666666\n",
+ "Average of max per entry across top 5 scores: 0.66\n",
+ "Average of max per entry across top 8 scores: 0.66\n",
+ "Average of max per entry across top 9999 scores: 0.66\n",
+ "9 candidate programs found.\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Prepare the training and validation sets for the optimizer using the original\n",
+ "# trainset. This ensures that our \"devset\" is left untouched.\n",
+ "shuffled_trainset = [d for d in trainset]\n",
+ "rng.shuffle(shuffled_trainset)\n",
+ "optimizer_trainset = shuffled_trainset[:OPTIMIZER_NUM_TRAIN]\n",
+ "optimizer_valset = shuffled_trainset[OPTIMIZER_NUM_TRAIN:OPTIMIZER_NUM_TRAIN+OPTIMIZER_NUM_VAL]\n",
+ "\n",
+ "# Initialize the optimizer\n",
+ "bfrs_optimizer = BootstrapFewShotWithRandomSearch(\n",
+ " metric=metric,\n",
+ " max_bootstrapped_demos=MAX_BOOTSTRAPPED_DEMOS,\n",
+ " max_labeled_demos=MAX_LABELED_DEMOS,\n",
+ " num_candidate_programs=NUM_CANDIDATE_PROGRAMS,\n",
+ " num_threads=NUM_THREADS\n",
+ ")\n",
+ "\n",
+ "# Compile the optimizer\n",
+ "with dspy.context(lm=lm, rm=retriever):\n",
+ " vanilla_program = BasicMH()\n",
+ " bfrs_base_program = bfrs_optimizer.compile(vanilla_program, trainset=optimizer_trainset, valset=optimizer_valset)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's evaluate our prompt optimized program on the base model!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Evaluating the prompt optimized program on the devset...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 262 / 500 (52.4): 100%|██████████| 500/500 [00:04<00:00, 115.45it/s]\n"
+ ]
+ }
+ ],
+ "source": [
+ "with dspy.context(lm=lm, rm=retriever):\n",
+ " print(\"Evaluating the prompt optimized program on the devset...\")\n",
+ " bfrs_base_eval = evaluate(bfrs_base_program)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## IV. Bootstrap Data"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In this section, we bootstrap data for fine-tuning.\n",
+ "In the code block below, we are deciding which program should be used to collect the bootstraps.\n",
+ "We are setting this to the prompt optimized program, but one could also set this to the vanilla program, though doing so would lead to lower quality bootstraps."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "bootstrap_program = bfrs_base_program # Can also set to vanilla_program"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We are now ready to bootstrap traces using our `trainset`!\n",
+ "The code block below does this using the `bootstrap_data` function, which takes in a program and a dataset; runs the program on the dataset and return collected data in dictionaries.\n",
+ "More information is included in the code comments below."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 540 / 1000 (54.0): 100%|██████████| 1000/1000 [00:07<00:00, 128.03it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "Trainset has 1000 examples.\n",
+ "Bootstrapped data has 1000 examples.\n",
+ "Bootstrapped data dictionaries have the following keys: ['example', 'prediction', 'trace', 'score']\n"
+ ]
+ }
+ ],
+ "source": [
+ "from dspy.teleprompt.finetune_teleprompter import bootstrap_data\n",
+ "\n",
+ "\n",
+ "# Bootstrap traces using the prompt optimized program\n",
+ "with dspy.context(lm=lm, rm=retriever):\n",
+ " bootstrapped_data = bootstrap_data(program=bootstrap_program, dataset=trainset, metric=metric, num_threads=NUM_THREADS)\n",
+ "\n",
+ "# The bootstra_data function returns a list of dictionaries, each dictionary\n",
+ "# corresponding to an exmaple in the data, with the the following keys and\n",
+ "# values:\n",
+ "# - The `example` field corresponding to the example itself\n",
+ "# - The `prediction` field corresponding to the prediction made by the program\n",
+ "# on the example.\n",
+ "# - The `trace` field corresponding to the trace generated by the program on the\n",
+ "# example.\n",
+ "# - The `score` field corresponding to the metric score of the example, if the\n",
+ "# metric is provided. Otherwise, it is not included in the data.\n",
+ "print(f\"\\nTrainset has {len(trainset)} examples.\")\n",
+ "print(f\"Bootstrapped data has {len(bootstrapped_data)} examples.\")\n",
+ "print(f\"Bootstrapped data dictionaries have the following keys: {list(bootstrapped_data[0].keys())}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Shuffle our data for training!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "rng.shuffle(bootstrapped_data)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We have now bootstrapped examples!\n",
+ "Before proceeding to fine-tuning, we filter out the unsuccessful examples, because we only want to keep the positive examples for our fine-tuning stage."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Filtered bootstrapped data has 540 examples.\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Filter the bootstrapped data to only include examples where the metric score\n",
+ "# is 1\n",
+ "filtered_bootstrapped_data = [d for d in bootstrapped_data if d['score']]\n",
+ "print(f\"Filtered bootstrapped data has {len(filtered_bootstrapped_data)} examples.\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Once we are settled on the particular bootstrap examples we want to use, we need to convert our bootstrap data (e.g. example traces) into prompt and completion \"pairs\" we could use for training.\n",
+ "To do so, we utilize the `trace` in the dictionaries returned by `bootstrap_data` function.\n",
+ "In particular, we create module level prompt and completion pairs, where the prompt correspond to any text passed to the model _initially_.\n",
+ "The completion then includes any intermediate generation (e.g. chain of though rationale) as well as the final outpus of the module (e.g. \"Answer:\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "For each datapoint, we get one prompt completion pair for each module in our program of 3 modules.\n",
+ "As a result, finetune data has 540 * 3 = 1620 examples.\n",
+ "The finetune data dictionaries have the following keys: ['prompt', 'completion', 'predictor_ind']\n"
+ ]
+ }
+ ],
+ "source": [
+ "from dspy.teleprompt.finetune_teleprompter import convert_to_module_level_prompt_completion_data\n",
+ "\n",
+ "# To train a model on our program traces, we need a dataset that consists of\n",
+ "# the prompts and the completions for each module in our program. We can use the\n",
+ "# helper method `convert_to_prompt_completion_data` to generate these traces.\n",
+ "# This function takes in a list of dictionaries, each of which must contain the\n",
+ "# \"trace\" field, which is used for generating the prompts and completions for\n",
+ "# each module in the program. The function returns a list of dictionaries, each\n",
+ "# of which contains the keys \"prompt\" and \"completion\", among others. Refer to\n",
+ "# the documentation for more information.\n",
+ "#\n",
+ "# The `exclude_demos` flag controls whether the demonstrations should be\n",
+ "# included in the prompts. We are setting it to True here to have the \"vanilla\"\n",
+ "# prompts as part of our finetuning data.\n",
+ "finetune_data = convert_to_module_level_prompt_completion_data(filtered_bootstrapped_data, exclude_demos=True)\n",
+ "print(f\"For each datapoint, we get one prompt completion pair for each module in our program of {len(vanilla_program.predictors())} modules.\")\n",
+ "print(f\"As a result, finetune data has {len(filtered_bootstrapped_data)} * {len(bootstrap_program.predictors())} = {len(finetune_data)} examples.\")\n",
+ "print(f\"The finetune data dictionaries have the following keys: {list(finetune_data[0].keys())}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's look at an example prompt completion pair!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "This prompt completion pair is collected from the following program using the predictor at index 0.\n",
+ "\n",
+ "generate_query[0] = Predict(StringSignature(context, question -> rationale, search_query\n",
+ " instructions='Given the fields `context`, `question`, produce the fields `search_query`.'\n",
+ " context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
+ " question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
+ " rationale = Field(annotation=str required=True json_schema_extra={'prefix': \"Reasoning: Let's think step by step in order to\", 'desc': '${produce the output fields}. We ...', '__dspy_field_type': 'output'})\n",
+ " search_query = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Search Query:', 'desc': '${search_query}'})\n",
+ "))\n",
+ "generate_query[1] = Predict(StringSignature(context, question -> rationale, search_query\n",
+ " instructions='Given the fields `context`, `question`, produce the fields `search_query`.'\n",
+ " context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
+ " question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
+ " rationale = Field(annotation=str required=True json_schema_extra={'prefix': \"Reasoning: Let's think step by step in order to\", 'desc': '${produce the output fields}. We ...', '__dspy_field_type': 'output'})\n",
+ " search_query = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Search Query:', 'desc': '${search_query}'})\n",
+ "))\n",
+ "generate_answer = Predict(StringSignature(context, question -> rationale, answer\n",
+ " instructions='Given the fields `context`, `question`, produce the fields `answer`.'\n",
+ " context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
+ " question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
+ " rationale = Field(annotation=str required=True json_schema_extra={'prefix': \"Reasoning: Let's think step by step in order to\", 'desc': '${produce the output fields}. We ...', '__dspy_field_type': 'output'})\n",
+ " answer = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Answer:', 'desc': '${answer}'})\n",
+ "))\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(f\"This prompt completion pair is collected from the following program using the predictor at index {finetune_data[0]['predictor_ind']}.\\n\")\n",
+ "print(bootstrap_program)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Given the fields `context`, `question`, produce the fields `search_query`.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the output fields}. We ...\n",
+ "\n",
+ "Search Query: ${search_query}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: N/A\n",
+ "\n",
+ "Question: Did Jeffrey Steele and Frank Iero both become Nashville songwriters?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(finetune_data[0]['prompt'])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ " determine if both Jeffrey Steele and Frank Iero became Nashville songwriters. First, I need to verify the careers of both individuals to see if they have worked as songwriters in Nashville. This involves looking up their professional backgrounds and any notable contributions they may have made to the Nashville music scene.\n",
+ "Search Query: Jeffrey Steele Frank Iero Nashville songwriters\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(finetune_data[0]['completion'])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Note that we don't shuffle this dataset again.\n",
+ "This leads to the prompt and complation pairs coming from the same traces to be co-located.\n",
+ "We now save our prompt completion data to a file, where each line is a separate dictionary.\n",
+ "This is the format expected by the fine-tuning methods in DSPy!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Writing dataset with length 1620 to trainset_data.jsonl\n"
+ ]
+ }
+ ],
+ "source": [
+ "import ujson\n",
+ "\n",
+ "train_path = \"trainset_data.jsonl\"\n",
+ "\n",
+ "print(f\"Writing dataset with length {len(finetune_data)} to {train_path}\")\n",
+ "with open(train_path, \"w\") as f:\n",
+ " ujson.dump(finetune_data, f)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## V. Fine-tune!"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We are now ready to start fine-tuning!\n",
+ "You can execute the next code block to get a fine-tuned model.\n",
+ "By default, you will get the reference to a model we trained.\n",
+ "You won't be able to make any new queries to it as `OpenAI` models aren't sharable publicly for the time being, but you can query it using the examples that are cached.\n",
+ "If you would like to train your own model, unset the `USE_CACHED_MODEL` flag.\n",
+ "\n",
+ "As mentioned, all the trainable LMs in DSPy implement the `dspy.TrainableLM` class.\n",
+ "The main interface to trainable LMs is the `get_finetune` method, which takes in a training method and a path to a train file, along with other kwargs that vary based on the specific subclass implementing the interface.\n",
+ "This method then returns a `future` object that holds a reference to the LM that's being fine-tuned.\n",
+ "Once the fine-tuning process completes, the `result` of this future object is populated with the fine-tuned model (or with any errors if the training didn't complete)\n",
+ "\n",
+ "Training from scratch took us around ~61 minutes - it may take you longer or shorter depending on how busy `OpenAI` servers are."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from dsp.modules.lm import TrainingMethod\n",
+ "\n",
+ "USE_CACHED_MODEL = False\n",
+ "\n",
+ "if USE_CACHED_MODEL:\n",
+ " CACHED_FINETUNED_MODEL_ID = \"\"\n",
+ " finetuned_lm = dspy.TrainableOpenAI(model=CACHED_FINETUNED_MODEL_ID, **MODEL_PARAMETERS)\n",
+ "else:\n",
+ " # Define the hyperparameters for finetuning. To look at the specifics of\n",
+ " # what kinds of arguments should be passed to `get_finetune`, refer to the\n",
+ " # documentation for the particular subclass you are using, in this case,\n",
+ " # `dspy.TrainableOpenAI`.\n",
+ " hyperparameters = {\n",
+ " \"n_epochs\": 3\n",
+ " }\n",
+ "\n",
+ " # Select the training method. Must be one of the options in the\n",
+ " # `TrainingMethod` enum.\n",
+ " method = TrainingMethod.SFT\n",
+ "\n",
+ " # Get a future object for the finetuned language model. This will get populated\n",
+ " # with the finetuned model once the training is complete.\n",
+ " future_finetuned_lm = lm.get_finetune(\n",
+ " method=method,\n",
+ " train_path=train_path,\n",
+ " hyperparameters=hyperparameters\n",
+ " )\n",
+ "\n",
+ " # Call the result method on the future object to get the finetuned language.\n",
+ " # This will block the execution until the training is complete.\n",
+ " finetuned_lm = future_finetuned_lm.result()\n",
+ "\n",
+ "# Record the model ID to access the model again in the future. You can directly\n",
+ "# substitute this model ID for the `CACHED_FINETUNED_MODEL_ID` variable above,\n",
+ "# and the next line will load the model directly.\n",
+ "finetuned_lm.kwargs['model']"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's evaluate our fine-tuned model!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "vanilla_program = BasicMH()\n",
+ "\n",
+ "with dspy.context(lm=finetuned_lm, rm=retriever):\n",
+ " print(\"Evaluating the vanilla program on the devset using the finetuned model...\")\n",
+ " vanilla_finetune_eval = evaluate(vanilla_program, devset=devset)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## VI. Prompt Optimize the Fine-tuned Model"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now that we have our fine-tuned model, we can prompt optimize it again!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Prepare the training and validation sets for the optimizer using the original\n",
+ "# trainset. This ensures that our \"devset\" is left untouched.\n",
+ "shuffled_trainset = [d for d in trainset]\n",
+ "rng.shuffle(shuffled_trainset)\n",
+ "optimizer_trainset = shuffled_trainset[:OPTIMIZER_NUM_TRAIN]\n",
+ "optimizer_valset = shuffled_trainset[OPTIMIZER_NUM_TRAIN:OPTIMIZER_NUM_TRAIN+OPTIMIZER_NUM_VAL]\n",
+ "\n",
+ "# Initialize the optimizer\n",
+ "bfrs_optimizer = BootstrapFewShotWithRandomSearch(\n",
+ " metric=metric,\n",
+ " max_bootstrapped_demos=MAX_BOOTSTRAPPED_DEMOS,\n",
+ " max_labeled_demos=MAX_LABELED_DEMOS,\n",
+ " num_candidate_programs=NUM_CANDIDATE_PROGRAMS,\n",
+ " num_threads=NUM_THREADS\n",
+ ")\n",
+ "\n",
+ "# Compile the optimizer and save the returned program\n",
+ "with dspy.context(lm=finetuned_lm, rm=retriever):\n",
+ " vanilla_program = BasicMH()\n",
+ " bfrs_finetune_program = bfrs_optimizer.compile(vanilla_program, trainset=optimizer_trainset, valset=optimizer_valset)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's see how well our prompt optimized fine-tuned model does!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "vanilla_program = BasicMH()\n",
+ "\n",
+ "with dspy.context(lm=finetuned_lm, rm=retriever):\n",
+ " print(\"Evaluating the vanilla program on the devset using the finetuned model...\")\n",
+ " bfrs_finetune_eval = evaluate(bfrs_finetune_program, devset=devset)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## VII. Summary"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The code block below summarizes all the results we had looked at!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "print(f\"Results for HotPotQA fine-tuning {GPT_4O_MINI} with a starting trainset with {len(trainset)}. Out of the {len(trainset)} examples, only {len(filtered_bootstrapped_data)} examples were judged as successful using a prompt optimized with BootstrapFewShotRandomSearch prompt optimized (bfrs) program on the base model. Traces from these examples are used for SFT. The reported results are computed on a held-out devset of {len(devset)} examples.\\n\")\n",
+ "print(f\" Base model (vanilla program): {vanilla_base_eval}\")\n",
+ "print(f\" Base model (bfrs program): {bfrs_base_eval}\")\n",
+ "print(f\" Fine-tuned model (vanilla program): {vanilla_finetune_eval}\")\n",
+ "print(f\" Fine-tuned model (bfrs program): {bfrs_finetune_eval}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "This was a demo notebook to showcase the experimental fine-tuning capabilities in DSPy!\n",
+ "We are actively working on making fine-tuning a first class citizen in DSPy -- do let us know if you have any suggestions!"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": ".venv",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.19"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}