DSPy: The framework for programming—not prompting—language models
Python
Update notebook
dilara52cd8fe39911
diff --git a/examples/finetuning/finetuning_demo.ipynb b/examples/finetuning/finetuning_demo.ipynbindex 5fe41ba8c..1a218af1f 100644--- a/examples/finetuning/finetuning_demo.ipynb+++ b/examples/finetuning/finetuning_demo.ipynb@@ -17,7 +17,8 @@"source": ["This is a demo notebook showing an example of how fine-tuning models work in DSPy.\n","In particular, we fine-tune `OpenAI`s `GPT-4o-mini` on `HotPotQA`.\n",- "Note that the fine-tuning features in DSPy are experimental and are subject to change."+ "Note that the fine-tuning features in DSPy are experimental and are subject to change.\n",+ "Running this notebook requires an `OPENAI_API_KEY`."]},{@@ -104,9 +105,7 @@"cell_type": "markdown","metadata": {},"source": [- "We are making use of a demo cache for this notebook, located in \"dspy/examples/finetuning/demo_cache\" in the `demo_finetuning` branch of the DSPy repository.\n",- "This allows us to utilize cached calls to the OpenAI API.\n",- "If you are not modifying this notebook, you don't need to set up the `OPENAI_API_KEY`."+ "We are using a separate cache for running the queries in this notebook - this is set in the next cell."]},{@@ -125,8 +124,8 @@"cell_type": "markdown","metadata": {},"source": [- "If you would like to modify the notebook and change the requests being made to the OpenAI API, you should set the `OPENAI_API_KEY`.\n",- "The next block demonstrates how."+ "Running this notebook requires an `OPENAI_API_KEY`.\n",+ "You can use one of the following methods to set it."]},{@@ -148,7 +147,10 @@"# line with the appropriate key. If you pick this option, make sure to\n","# remove your key before sharing your notebook or committing to a\n","# repository.\n",- "# os.environ[\"OPENAI_API_KEY\"] = <OPENAI_API_KEY>"+ "# os.environ[\"OPENAI_API_KEY\"] = <OPENAI_API_KEY>\n",+ "\n",+ "# Ensure that the OPENAI_API_KEY is set before continuing\n",+ "assert os.environ[\"OPENAI_API_KEY\"]"]},{@@ -1117,69 +1119,7 @@"cell_type": "markdown","metadata": {},"source": [- "## VI. Prompt Optimize the Fine-tuned Model"- ]- },- {- "cell_type": "markdown",- "metadata": {},- "source": [- "Now that we have our fine-tuned model, we can prompt optimize it again!"- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "# Prepare the training and validation sets for the optimizer using the original\n",- "# trainset. This ensures that our \"devset\" is left untouched.\n",- "shuffled_trainset = [d for d in trainset]\n",- "rng.shuffle(shuffled_trainset)\n",- "optimizer_trainset = shuffled_trainset[:OPTIMIZER_NUM_TRAIN]\n",- "optimizer_valset = shuffled_trainset[OPTIMIZER_NUM_TRAIN:OPTIMIZER_NUM_TRAIN+OPTIMIZER_NUM_VAL]\n",- "\n",- "# Initialize the optimizer\n",- "bfrs_optimizer = BootstrapFewShotWithRandomSearch(\n",- " metric=metric,\n",- " max_bootstrapped_demos=MAX_BOOTSTRAPPED_DEMOS,\n",- " max_labeled_demos=MAX_LABELED_DEMOS,\n",- " num_candidate_programs=NUM_CANDIDATE_PROGRAMS,\n",- " num_threads=NUM_THREADS\n",- ")\n",- "\n",- "# Compile the optimizer and save the returned program\n",- "with dspy.context(lm=finetuned_lm, rm=retriever):\n",- " vanilla_program = BasicMH()\n",- " bfrs_finetune_program = bfrs_optimizer.compile(vanilla_program, trainset=optimizer_trainset, valset=optimizer_valset)"- ]- },- {- "cell_type": "markdown",- "metadata": {},- "source": [- "Let's see how well our prompt optimized fine-tuned model does!"- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "vanilla_program = BasicMH()\n",- "\n",- "with dspy.context(lm=finetuned_lm, rm=retriever):\n",- " print(\"Evaluating the vanilla program on the devset using the finetuned model...\")\n",- " bfrs_finetune_eval = evaluate(bfrs_finetune_program, devset=devset)"- ]- },- {- "cell_type": "markdown",- "metadata": {},- "source": [- "## VII. Summary"+ "## VI. Summary"]},{@@ -1198,8 +1138,7 @@"print(f\"Results for HotPotQA fine-tuning {GPT_4O_MINI} with a starting trainset with {len(trainset)}. Out of the {len(trainset)} examples, only {len(filtered_bootstrapped_data)} examples were judged as successful using a prompt optimized with BootstrapFewShotRandomSearch prompt optimized (bfrs) program on the base model. Traces from these examples are used for SFT. The reported results are computed on a held-out devset of {len(devset)} examples.\\n\")\n","print(f\" Base model (vanilla program): {vanilla_base_eval}\")\n","print(f\" Base model (bfrs program): {bfrs_base_eval}\")\n",- "print(f\" Fine-tuned model (vanilla program): {vanilla_finetune_eval}\")\n",- "print(f\" Fine-tuned model (bfrs program): {bfrs_finetune_eval}\")"+ "print(f\" Fine-tuned model (vanilla program): {vanilla_finetune_eval}\")"]},{