Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Update notebook

dilara52cd8fe39911
diff --git a/examples/finetuning/finetuning_demo.ipynb b/examples/finetuning/finetuning_demo.ipynb
index 5fe41ba8c..1a218af1f 100644
--- a/examples/finetuning/finetuning_demo.ipynb
+++ b/examples/finetuning/finetuning_demo.ipynb
@@ -17,7 +17,8 @@
"source": [
"This is a demo notebook showing an example of how fine-tuning models work in DSPy.\n",
"In particular, we fine-tune `OpenAI`s `GPT-4o-mini` on `HotPotQA`.\n",
- "Note that the fine-tuning features in DSPy are experimental and are subject to change."
+ "Note that the fine-tuning features in DSPy are experimental and are subject to change.\n",
+ "Running this notebook requires an `OPENAI_API_KEY`."
]
},
{
@@ -104,9 +105,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "We are making use of a demo cache for this notebook, located in \"dspy/examples/finetuning/demo_cache\" in the `demo_finetuning` branch of the DSPy repository.\n",
- "This allows us to utilize cached calls to the OpenAI API.\n",
- "If you are not modifying this notebook, you don't need to set up the `OPENAI_API_KEY`."
+ "We are using a separate cache for running the queries in this notebook - this is set in the next cell."
]
},
{
@@ -125,8 +124,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "If you would like to modify the notebook and change the requests being made to the OpenAI API, you should set the `OPENAI_API_KEY`.\n",
- "The next block demonstrates how."
+ "Running this notebook requires an `OPENAI_API_KEY`.\n",
+ "You can use one of the following methods to set it."
]
},
{
@@ -148,7 +147,10 @@
"# line with the appropriate key. If you pick this option, make sure to\n",
"# remove your key before sharing your notebook or committing to a\n",
"# repository.\n",
- "# os.environ[\"OPENAI_API_KEY\"] = <OPENAI_API_KEY>"
+ "# os.environ[\"OPENAI_API_KEY\"] = <OPENAI_API_KEY>\n",
+ "\n",
+ "# Ensure that the OPENAI_API_KEY is set before continuing\n",
+ "assert os.environ[\"OPENAI_API_KEY\"]"
]
},
{
@@ -1117,69 +1119,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## VI. Prompt Optimize the Fine-tuned Model"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now that we have our fine-tuned model, we can prompt optimize it again!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Prepare the training and validation sets for the optimizer using the original\n",
- "# trainset. This ensures that our \"devset\" is left untouched.\n",
- "shuffled_trainset = [d for d in trainset]\n",
- "rng.shuffle(shuffled_trainset)\n",
- "optimizer_trainset = shuffled_trainset[:OPTIMIZER_NUM_TRAIN]\n",
- "optimizer_valset = shuffled_trainset[OPTIMIZER_NUM_TRAIN:OPTIMIZER_NUM_TRAIN+OPTIMIZER_NUM_VAL]\n",
- "\n",
- "# Initialize the optimizer\n",
- "bfrs_optimizer = BootstrapFewShotWithRandomSearch(\n",
- " metric=metric,\n",
- " max_bootstrapped_demos=MAX_BOOTSTRAPPED_DEMOS,\n",
- " max_labeled_demos=MAX_LABELED_DEMOS,\n",
- " num_candidate_programs=NUM_CANDIDATE_PROGRAMS,\n",
- " num_threads=NUM_THREADS\n",
- ")\n",
- "\n",
- "# Compile the optimizer and save the returned program\n",
- "with dspy.context(lm=finetuned_lm, rm=retriever):\n",
- " vanilla_program = BasicMH()\n",
- " bfrs_finetune_program = bfrs_optimizer.compile(vanilla_program, trainset=optimizer_trainset, valset=optimizer_valset)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's see how well our prompt optimized fine-tuned model does!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "vanilla_program = BasicMH()\n",
- "\n",
- "with dspy.context(lm=finetuned_lm, rm=retriever):\n",
- " print(\"Evaluating the vanilla program on the devset using the finetuned model...\")\n",
- " bfrs_finetune_eval = evaluate(bfrs_finetune_program, devset=devset)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## VII. Summary"
+ "## VI. Summary"
]
},
{
@@ -1198,8 +1138,7 @@
"print(f\"Results for HotPotQA fine-tuning {GPT_4O_MINI} with a starting trainset with {len(trainset)}. Out of the {len(trainset)} examples, only {len(filtered_bootstrapped_data)} examples were judged as successful using a prompt optimized with BootstrapFewShotRandomSearch prompt optimized (bfrs) program on the base model. Traces from these examples are used for SFT. The reported results are computed on a held-out devset of {len(devset)} examples.\\n\")\n",
"print(f\" Base model (vanilla program): {vanilla_base_eval}\")\n",
"print(f\" Base model (bfrs program): {bfrs_base_eval}\")\n",
- "print(f\" Fine-tuned model (vanilla program): {vanilla_finetune_eval}\")\n",
- "print(f\" Fine-tuned model (bfrs program): {bfrs_finetune_eval}\")"
+ "print(f\" Fine-tuned model (vanilla program): {vanilla_finetune_eval}\")"
]
},
{