DSPy: The framework for programming—not prompting—language models
Python
Local finetuning tutorial (#2024)
Omar Khattab9c5a44ec2a3e
diff --git a/docs/docs/tutorials/classification_finetuning/index.ipynb b/docs/docs/tutorials/classification_finetuning/index.ipynbnew file mode 100644index 000000000..0e103e9b1--- /dev/null+++ b/docs/docs/tutorials/classification_finetuning/index.ipynb@@ -0,0 +1,970 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "# Tutorial: Classification Fine-tuning\n",+ "\n",+ "Let's walk through a quick example of fine-tuning the LM weights within a DSPy program. We'll apply to a simple 77-way classification task.\n",+ "\n",+ "Our finetuned program will use a tiny `Llama-3.2-1B` language model, hosted locally on your GPU. To make this more interesting, we'll assume that (i) we don't have **any training labels** but (ii) we have 500 unlabeled training examples.\n",+ "\n",+ "### Install dependencies and download data\n",+ "\n",+ "Install the latest DSPy via `pip install -U --pre dspy` and follow along. This tutorial depends on DSPy 2.6.0 (pre-release).\n",+ "\n",+ "This tutorial requires a local GPU at the moment for inference, though we plan to support ollama serving for finetuned models as well.\n",+ "\n",+ "You will also need the following dependencies:\n",+ "\n",+ "```shell\n",+ "> pip install \"sglang[all]\"; pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/\n",+ "> pip install -U torch transformers accelerate trl peft\n",+ "```"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Dataset\n",+ "\n",+ "For this tutorial, we will use the Banking77 dataset."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 1,+ "metadata": {},+ "outputs": [],+ "source": [+ "import dspy\n",+ "import random\n",+ "from dspy.datasets import DataLoader\n",+ "from datasets import load_dataset\n",+ "\n",+ "# Load the Banking77 dataset.\n",+ "CLASSES = load_dataset(\"PolyAI/banking77\", split=\"train\", trust_remote_code=True).features['label'].names\n",+ "kwargs = dict(fields=(\"text\", \"label\"), input_keys=(\"text\",), split=\"train\", trust_remote_code=True)\n",+ "\n",+ "# Load the first 2000 examples from the dataset, and assign a hint to each *training* example.\n",+ "raw_data = [\n",+ " dspy.Example(x, label=CLASSES[x.label]).with_inputs(\"text\")\n",+ " for x in DataLoader().from_huggingface(dataset_name=\"PolyAI/banking77\", **kwargs)[:1000]\n",+ "]\n",+ "\n",+ "random.Random(0).shuffle(raw_data)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "This dataset has 77 different categories for classification. Let's review some of them."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 2,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "(77,\n",+ " ['activate_my_card',\n",+ " 'age_limit',\n",+ " 'apple_pay_or_google_pay',\n",+ " 'atm_support',\n",+ " 'automatic_top_up',\n",+ " 'balance_not_updated_after_bank_transfer',\n",+ " 'balance_not_updated_after_cheque_or_cash_deposit',\n",+ " 'beneficiary_not_allowed',\n",+ " 'cancel_transfer',\n",+ " 'card_about_to_expire'])"+ ]+ },+ "execution_count": 2,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "len(CLASSES), CLASSES[:10]"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Let us sample 500 (unlabeled) queries from Banking77. We'll use these for our bootstrapped finetuning."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 3,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "Example({'text': 'What if there is an error on the exchange rate?'}) (input_keys={'text'})"+ ]+ },+ "execution_count": 3,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "unlabeled_trainset = [dspy.Example(text=x.text).with_inputs(\"text\") for x in raw_data[:500]]\n",+ "\n",+ "unlabeled_trainset[0]"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### DSPy program\n",+ "\n",+ "Let's say that we want a program that takes the `text` and reasons step by step and then selects one of the classes from Banking77.\n",+ "\n",+ "Note that this is meant mainly for illustration, or for cases where you want to inspect the model's reasoning, e.g. for a small degree of explainability. In other words, this type of task is not necessarily likely to benefit very much from explicit reasoning."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 4,+ "metadata": {},+ "outputs": [],+ "source": [+ "from typing import Literal\n",+ "\n",+ "classify = dspy.ChainOfThought(f\"text -> label: Literal{CLASSES}\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Bootstrapped finetuning\n",+ "\n",+ "There are many ways to go about this, e.g. allowing the model to teach itself or using inference-time compute (e.g., ensembling) to identify cases of high confidence without labels.\n",+ "\n",+ "Perhaps the simplest is to use a model that we'd expect can do a reasonable job at this task as a teacher of reasoning and classification, and to distill that to our small model. All of these patterns can be expressed in a handful of lines of code.\n",+ "\n",+ "Let's set up the tiny `Llama-3.2-1B-Instruct` as a student LM. We'll use GPT-4o-mini as a teacher LM."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 5,+ "metadata": {},+ "outputs": [],+ "source": [+ "from dspy.clients.lm_local import LocalProvider\n",+ "\n",+ "student_lm_name = \"meta-llama/Llama-3.2-1B-Instruct\"\n",+ "student_lm = dspy.LM(model=f\"openai/local:{student_lm_name}\", provider=LocalProvider(), max_tokens=2000)\n",+ "teacher_lm = dspy.LM('openai/gpt-4o-mini', max_tokens=3000)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Now, let's assign classifiers to our LMs."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 6,+ "metadata": {},+ "outputs": [],+ "source": [+ "student_classify = classify.deepcopy()\n",+ "student_classify.set_lm(student_lm)\n",+ "\n",+ "teacher_classify = classify.deepcopy()\n",+ "teacher_classify.set_lm(teacher_lm)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Let's now launch the bootstrapped finetuning. The word \"bootstrapped\" here means that the program itself will be invoked on the training inputs and the resulting traces seen over all modules will be recorded and used for finetuning. This is the weight-optimizing variant of the various BootstrapFewShot methods in DSPy.\n",+ "\n",+ "On every question in the (unlabeled) training set, this will invoke the teacher program, which will produce reasoning and select a class. This will be traced and then constitute a training set for all modules (in this case, just the one CoT module) in the student program.\n",+ "\n",+ "Note: If you have labels, you can pass `metric` to the constructor of `BootstrapFinetune`. If you want to apply this in practice, you can pass `train_kwargs` to the constructor to control local LM training settings: `device`, `use_peft`, `num_train_epochs`, `per_device_train_batch_size`, `gradient_accumulation_steps`, `learning_rate`, `max_seq_length`, `packing`, `bf16`, and `output_dir`."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 7,+ "metadata": {},+ "outputs": [],+ "source": [+ "optimizer = dspy.BootstrapFinetune(num_threads=16) # if you *do* have labels, pass metric=your_metric here!\n",+ "classify_ft = optimizer.compile(student_classify, teacher=teacher_classify, trainset=unlabeled_trainset)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Since this is a local model, we need to explicitly launch it."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 8,+ "metadata": {},+ "outputs": [],+ "source": [+ "classify_ft.get_lm().launch()"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Validating the finetuned program\n",+ "\n",+ "Let's now figure out if this was successful. We can ask the system one question and inspect its behavior."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 9,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "Prediction(\n",+ " reasoning='The user is inquiring about a specific issue, which they did not receive and is still showing as a pending transaction. This situation typically indicates a problem with the cash withdrawal process, as the user is not receiving the money they attempted to withdraw. The appropriate label for this scenario is \"pending_cash_withdrawal,\" as it directly relates to the status of the cash withdrawal transaction.',\n",+ " label='pending_cash_withdrawal'\n",+ ")"+ ]+ },+ "execution_count": 9,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "classify_ft(text=\"I didn't receive my money earlier and it says the transaction is still in progress. Can you fix it?\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "We could also get a small set of gold labels and see if the system can generalize to unseen queries."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 10,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "Example({'text': 'Which fiat currencies do you currently support? Will this change in this future?', 'label': 'fiat_currency_support'}) (input_keys={'text'})"+ ]+ },+ "execution_count": 10,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "devset = raw_data[500:600]\n",+ "devset[0]"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Let's define an evaluator on this small dev set, where the metric ignores the reasoning and checks that the label is exactly correct."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 11,+ "metadata": {},+ "outputs": [],+ "source": [+ "metric = (lambda x, y, trace=None: x.label == y.label)\n",+ "evaluate = dspy.Evaluate(devset=devset, metric=metric, display_progress=True, display_table=5, num_threads=16)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Now, let's evaluate the finetuned 1B classifier."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 12,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Average Metric: 51.00 / 99 (51.5%): 100%|██████████| 100/100 [00:35<00:00, 2.79it/s]"+ ]+ },+ {+ "data": {+ "text/html": [+ "<div>\n",+ "<style scoped>\n",+ " .dataframe tbody tr th:only-of-type {\n",+ " vertical-align: middle;\n",+ " }\n",+ "\n",+ " .dataframe tbody tr th {\n",+ " vertical-align: top;\n",+ " }\n",+ "\n",+ " .dataframe thead th {\n",+ " text-align: right;\n",+ " }\n",+ "</style>\n",+ "<table border=\"1\" class=\"dataframe\">\n",+ " <thead>\n",+ " <tr style=\"text-align: right;\">\n",+ " <th></th>\n",+ " <th>text</th>\n",+ " <th>example_label</th>\n",+ " <th>reasoning</th>\n",+ " <th>pred_label</th>\n",+ " <th><lambda></th>\n",+ " <th>label</th>\n",+ " </tr>\n",+ " </thead>\n",+ " <tbody>\n",+ " <tr>\n",+ " <th>0</th>\n",+ " <td>Which fiat currencies do you currently support? Will this change i...</td>\n",+ " <td>fiat_currency_support</td>\n",+ " <td>The user is inquiring about the current support for fiat currencie...</td>\n",+ " <td>fiat_currency_support</td>\n",+ " <td>✔️ [True]</td>\n",+ " <td>NaN</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>1</th>\n",+ " <td>I didn't receive my money earlier and it says the transaction is s...</td>\n",+ " <td>pending_cash_withdrawal</td>\n",+ " <td>The user is inquiring about a specific issue, which they did not r...</td>\n",+ " <td>pending_cash_withdrawal</td>\n",+ " <td>✔️ [True]</td>\n",+ " <td>NaN</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>2</th>\n",+ " <td>what currencies do you accept?</td>\n",+ " <td>fiat_currency_support</td>\n",+ " <td>The user is inquiring about the currencies that are accepted, whic...</td>\n",+ " <td>fiat_currency_support</td>\n",+ " <td>✔️ [True]</td>\n",+ " <td>NaN</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>3</th>\n",+ " <td>Where can I find your exchange rates?</td>\n",+ " <td>exchange_rate</td>\n",+ " <td>The user is inquiring about where to find exchange rates, which re...</td>\n",+ " <td>exchange_rate</td>\n",+ " <td>✔️ [True]</td>\n",+ " <td>NaN</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>4</th>\n",+ " <td>why hasnt my card come in yet?</td>\n",+ " <td>card_arrival</td>\n",+ " <td>The user is inquiring about the status of their card, which sugges...</td>\n",+ " <td>card_arrival</td>\n",+ " <td>✔️ [True]</td>\n",+ " <td>NaN</td>\n",+ " </tr>\n",+ " </tbody>\n",+ "</table>\n",+ "</div>"+ ],+ "text/plain": [+ " text \\\n",+ "0 Which fiat currencies do you currently support? Will this change i... \n",+ "1 I didn't receive my money earlier and it says the transaction is s... \n",+ "2 what currencies do you accept? \n",+ "3 Where can I find your exchange rates? \n",+ "4 why hasnt my card come in yet? \n",+ "\n",+ " example_label \\\n",+ "0 fiat_currency_support \n",+ "1 pending_cash_withdrawal \n",+ "2 fiat_currency_support \n",+ "3 exchange_rate \n",+ "4 card_arrival \n",+ "\n",+ " reasoning \\\n",+ "0 The user is inquiring about the current support for fiat currencie... \n",+ "1 The user is inquiring about a specific issue, which they did not r... \n",+ "2 The user is inquiring about the currencies that are accepted, whic... \n",+ "3 The user is inquiring about where to find exchange rates, which re... \n",+ "4 The user is inquiring about the status of their card, which sugges... \n",+ "\n",+ " pred_label <lambda> label \n",+ "0 fiat_currency_support ✔️ [True] NaN \n",+ "1 pending_cash_withdrawal ✔️ [True] NaN \n",+ "2 fiat_currency_support ✔️ [True] NaN \n",+ "3 exchange_rate ✔️ [True] NaN \n",+ "4 card_arrival ✔️ [True] NaN "+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ },+ {+ "data": {+ "text/html": [+ "\n",+ " <div style='\n",+ " text-align: center;\n",+ " font-size: 16px;\n",+ " font-weight: bold;\n",+ " color: #555;\n",+ " margin: 10px 0;'>\n",+ " ... 95 more rows not displayed ...\n",+ " </div>\n",+ " "+ ],+ "text/plain": [+ "<IPython.core.display.HTML object>"+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ },+ {+ "data": {+ "text/plain": [+ "51.0"+ ]+ },+ "execution_count": 13,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "evaluate(classify_ft)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Not bad, given that we started with no labels of the task. Even though we have no labels, you can use various strategies to boost the quality of the bootstrapped training data.\n",+ "\n",+ "To try that next, let's free our GPU memory by killing the finetuned LM."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 14,+ "metadata": {},+ "outputs": [],+ "source": [+ "classify_ft.get_lm().kill()"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Bootstrapped finetuning against a metric\n",+ "\n",+ "If you have labels, you can generally boost this by a large margin. To do so, you can pass a `metric` to BootstrapFinetune, which it will use for filtering the trajectories over your program before it builds the finetuning data."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 15,+ "metadata": {},+ "outputs": [],+ "source": [+ "optimizer = dspy.BootstrapFinetune(num_threads=16, metric=metric)\n",+ "classify_ft = optimizer.compile(student_classify, teacher=teacher_classify, trainset=raw_data[:500])"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Let's now launch and evaluate this."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 16,+ "metadata": {},+ "outputs": [],+ "source": [+ "classify_ft.get_lm().launch()"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 17,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Average Metric: 85.00 / 98 (86.7%): 100%|██████████| 100/100 [00:46<00:00, 2.14it/s]"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "\n"+ ]+ },+ {+ "data": {+ "text/html": [+ "<div>\n",+ "<style scoped>\n",+ " .dataframe tbody tr th:only-of-type {\n",+ " vertical-align: middle;\n",+ " }\n",+ "\n",+ " .dataframe tbody tr th {\n",+ " vertical-align: top;\n",+ " }\n",+ "\n",+ " .dataframe thead th {\n",+ " text-align: right;\n",+ " }\n",+ "</style>\n",+ "<table border=\"1\" class=\"dataframe\">\n",+ " <thead>\n",+ " <tr style=\"text-align: right;\">\n",+ " <th></th>\n",+ " <th>text</th>\n",+ " <th>example_label</th>\n",+ " <th>reasoning</th>\n",+ " <th>pred_label</th>\n",+ " <th><lambda></th>\n",+ " <th>label</th>\n",+ " </tr>\n",+ " </thead>\n",+ " <tbody>\n",+ " <tr>\n",+ " <th>0</th>\n",+ " <td>Which fiat currencies do you currently support? Will this change i...</td>\n",+ " <td>fiat_currency_support</td>\n",+ " <td>The user is inquiring about the fiat currencies currently supporte...</td>\n",+ " <td>fiat_currency_support</td>\n",+ " <td>✔️ [True]</td>\n",+ " <td>NaN</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>1</th>\n",+ " <td>I didn't receive my money earlier and it says the transaction is s...</td>\n",+ " <td>pending_cash_withdrawal</td>\n",+ " <td>The user is inquiring about an unexpected fee on their account, wh...</td>\n",+ " <td>extra_charge_on_statement</td>\n",+ " <td></td>\n",+ " <td>NaN</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>2</th>\n",+ " <td>what currencies do you accept?</td>\n",+ " <td>fiat_currency_support</td>\n",+ " <td>The user is inquiring about the types of currencies that are accep...</td>\n",+ " <td>fiat_currency_support</td>\n",+ " <td>✔️ [True]</td>\n",+ " <td>NaN</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>3</th>\n",+ " <td>Where can I find your exchange rates?</td>\n",+ " <td>exchange_rate</td>\n",+ " <td>The user is inquiring about where to find exchange rates, which re...</td>\n",+ " <td>exchange_rate</td>\n",+ " <td>✔️ [True]</td>\n",+ " <td>NaN</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>4</th>\n",+ " <td>why hasnt my card come in yet?</td>\n",+ " <td>card_arrival</td>\n",+ " <td>The user is inquiring about the status of their card delivery, whi...</td>\n",+ " <td>card_arrival</td>\n",+ " <td>✔️ [True]</td>\n",+ " <td>NaN</td>\n",+ " </tr>\n",+ " </tbody>\n",+ "</table>\n",+ "</div>"+ ],+ "text/plain": [+ " text \\\n",+ "0 Which fiat currencies do you currently support? Will this change i... \n",+ "1 I didn't receive my money earlier and it says the transaction is s... \n",+ "2 what currencies do you accept? \n",+ "3 Where can I find your exchange rates? \n",+ "4 why hasnt my card come in yet? \n",+ "\n",+ " example_label \\\n",+ "0 fiat_currency_support \n",+ "1 pending_cash_withdrawal \n",+ "2 fiat_currency_support \n",+ "3 exchange_rate \n",+ "4 card_arrival \n",+ "\n",+ " reasoning \\\n",+ "0 The user is inquiring about the fiat currencies currently supporte... \n",+ "1 The user is inquiring about an unexpected fee on their account, wh... \n",+ "2 The user is inquiring about the types of currencies that are accep... \n",+ "3 The user is inquiring about where to find exchange rates, which re... \n",+ "4 The user is inquiring about the status of their card delivery, whi... \n",+ "\n",+ " pred_label <lambda> label \n",+ "0 fiat_currency_support ✔️ [True] NaN \n",+ "1 extra_charge_on_statement NaN \n",+ "2 fiat_currency_support ✔️ [True] NaN \n",+ "3 exchange_rate ✔️ [True] NaN \n",+ "4 card_arrival ✔️ [True] NaN "+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ },+ {+ "data": {+ "text/html": [+ "\n",+ " <div style='\n",+ " text-align: center;\n",+ " font-size: 16px;\n",+ " font-weight: bold;\n",+ " color: #555;\n",+ " margin: 10px 0;'>\n",+ " ... 95 more rows not displayed ...\n",+ " </div>\n",+ " "+ ],+ "text/plain": [+ "<IPython.core.display.HTML object>"+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ },+ {+ "data": {+ "text/plain": [+ "85.0"+ ]+ },+ "execution_count": 29,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "evaluate(classify_ft)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "That's quite a bit better, given just 500 labels. In fact, it seems to be a lot stronger than the teacher LM gets out of the box!"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 30,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Average Metric: 55.00 / 100 (55.0%): 100%|██████████| 100/100 [00:11<00:00, 8.88it/s]"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "2025/01/08 12:38:35 INFO dspy.evaluate.evaluate: Average Metric: 55 / 100 (55.0%)\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "\n"+ ]+ },+ {+ "data": {+ "text/html": [+ "<div>\n",+ "<style scoped>\n",+ " .dataframe tbody tr th:only-of-type {\n",+ " vertical-align: middle;\n",+ " }\n",+ "\n",+ " .dataframe tbody tr th {\n",+ " vertical-align: top;\n",+ " }\n",+ "\n",+ " .dataframe thead th {\n",+ " text-align: right;\n",+ " }\n",+ "</style>\n",+ "<table border=\"1\" class=\"dataframe\">\n",+ " <thead>\n",+ " <tr style=\"text-align: right;\">\n",+ " <th></th>\n",+ " <th>text</th>\n",+ " <th>example_label</th>\n",+ " <th>reasoning</th>\n",+ " <th>pred_label</th>\n",+ " <th><lambda></th>\n",+ " </tr>\n",+ " </thead>\n",+ " <tbody>\n",+ " <tr>\n",+ " <th>0</th>\n",+ " <td>Which fiat currencies do you currently support? Will this change i...</td>\n",+ " <td>fiat_currency_support</td>\n",+ " <td>The user is inquiring about the fiat currencies supported by the s...</td>\n",+ " <td>fiat_currency_support</td>\n",+ " <td>✔️ [True]</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>1</th>\n",+ " <td>I didn't receive my money earlier and it says the transaction is s...</td>\n",+ " <td>pending_cash_withdrawal</td>\n",+ " <td>The user is experiencing an issue with a transaction that is still...</td>\n",+ " <td>pending_transfer</td>\n",+ " <td></td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>2</th>\n",+ " <td>what currencies do you accept?</td>\n",+ " <td>fiat_currency_support</td>\n",+ " <td>The question is asking about the types of currencies accepted, whi...</td>\n",+ " <td>fiat_currency_support</td>\n",+ " <td>✔️ [True]</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>3</th>\n",+ " <td>Where can I find your exchange rates?</td>\n",+ " <td>exchange_rate</td>\n",+ " <td>The user is inquiring about where to find exchange rates, which re...</td>\n",+ " <td>exchange_rate</td>\n",+ " <td>✔️ [True]</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>4</th>\n",+ " <td>why hasnt my card come in yet?</td>\n",+ " <td>card_arrival</td>\n",+ " <td>The user is inquiring about the status of their card delivery, whi...</td>\n",+ " <td>card_delivery_estimate</td>\n",+ " <td></td>\n",+ " </tr>\n",+ " </tbody>\n",+ "</table>\n",+ "</div>"+ ],+ "text/plain": [+ " text \\\n",+ "0 Which fiat currencies do you currently support? Will this change i... \n",+ "1 I didn't receive my money earlier and it says the transaction is s... \n",+ "2 what currencies do you accept? \n",+ "3 Where can I find your exchange rates? \n",+ "4 why hasnt my card come in yet? \n",+ "\n",+ " example_label \\\n",+ "0 fiat_currency_support \n",+ "1 pending_cash_withdrawal \n",+ "2 fiat_currency_support \n",+ "3 exchange_rate \n",+ "4 card_arrival \n",+ "\n",+ " reasoning \\\n",+ "0 The user is inquiring about the fiat currencies supported by the s... \n",+ "1 The user is experiencing an issue with a transaction that is still... \n",+ "2 The question is asking about the types of currencies accepted, whi... \n",+ "3 The user is inquiring about where to find exchange rates, which re... \n",+ "4 The user is inquiring about the status of their card delivery, whi... \n",+ "\n",+ " pred_label <lambda> \n",+ "0 fiat_currency_support ✔️ [True] \n",+ "1 pending_transfer \n",+ "2 fiat_currency_support ✔️ [True] \n",+ "3 exchange_rate ✔️ [True] \n",+ "4 card_delivery_estimate "+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ },+ {+ "data": {+ "text/html": [+ "\n",+ " <div style='\n",+ " text-align: center;\n",+ " font-size: 16px;\n",+ " font-weight: bold;\n",+ " color: #555;\n",+ " margin: 10px 0;'>\n",+ " ... 95 more rows not displayed ...\n",+ " </div>\n",+ " "+ ],+ "text/plain": [+ "<IPython.core.display.HTML object>"+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ },+ {+ "data": {+ "text/plain": [+ "55.0"+ ]+ },+ "execution_count": 17,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "evaluate(teacher_classify)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "And thanks to bootstrapping, the model learns to apply our modules to get the right label, in this case, reasoning explicitly:"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 18,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "\n",+ "\n",+ "\n",+ "\n",+ "\u001b[34m[2025-01-08T12:39:42.143798]\u001b[0m\n",+ "\n",+ "\u001b[31mSystem message:\u001b[0m\n",+ "\n",+ "Your input fields are:\n",+ "1. `text` (str)\n",+ "\n",+ "Your output fields are:\n",+ "1. `reasoning` (str)\n",+ "2. `label` (Literal[activate_my_card, age_limit, apple_pay_or_google_pay, atm_support, automatic_top_up, balance_not_updated_after_bank_transfer, balance_not_updated_after_cheque_or_cash_deposit, beneficiary_not_allowed, cancel_transfer, card_about_to_expire, card_acceptance, card_arrival, card_delivery_estimate, card_linking, card_not_working, card_payment_fee_charged, card_payment_not_recognised, card_payment_wrong_exchange_rate, card_swallowed, cash_withdrawal_charge, cash_withdrawal_not_recognised, change_pin, compromised_card, contactless_not_working, country_support, declined_card_payment, declined_cash_withdrawal, declined_transfer, direct_debit_payment_not_recognised, disposable_card_limits, edit_personal_details, exchange_charge, exchange_rate, exchange_via_app, extra_charge_on_statement, failed_transfer, fiat_currency_support, get_disposable_virtual_card, get_physical_card, getting_spare_card, getting_virtual_card, lost_or_stolen_card, lost_or_stolen_phone, order_physical_card, passcode_forgotten, pending_card_payment, pending_cash_withdrawal, pending_top_up, pending_transfer, pin_blocked, receiving_money, Refund_not_showing_up, request_refund, reverted_card_payment?, supported_cards_and_currencies, terminate_account, top_up_by_bank_transfer_charge, top_up_by_card_charge, top_up_by_cash_or_cheque, top_up_failed, top_up_limits, top_up_reverted, topping_up_by_card, transaction_charged_twice, transfer_fee_charged, transfer_into_account, transfer_not_received_by_recipient, transfer_timing, unable_to_verify_identity, verify_my_identity, verify_source_of_funds, verify_top_up, virtual_card_not_working, visa_or_mastercard, why_verify_identity, wrong_amount_of_cash_received, wrong_exchange_rate_for_cash_withdrawal])\n",+ "\n",+ "All interactions will be structured in the following way, with the appropriate values filled in.\n",+ "\n",+ "[[ ## text ## ]]\n",+ "{text}\n",+ "\n",+ "[[ ## reasoning ## ]]\n",+ "{reasoning}\n",+ "\n",+ "[[ ## label ## ]]\n",+ "{label} # note: the value you produce must be one of: activate_my_card; age_limit; apple_pay_or_google_pay; atm_support; automatic_top_up; balance_not_updated_after_bank_transfer; balance_not_updated_after_cheque_or_cash_deposit; beneficiary_not_allowed; cancel_transfer; card_about_to_expire; card_acceptance; card_arrival; card_delivery_estimate; card_linking; card_not_working; card_payment_fee_charged; card_payment_not_recognised; card_payment_wrong_exchange_rate; card_swallowed; cash_withdrawal_charge; cash_withdrawal_not_recognised; change_pin; compromised_card; contactless_not_working; country_support; declined_card_payment; declined_cash_withdrawal; declined_transfer; direct_debit_payment_not_recognised; disposable_card_limits; edit_personal_details; exchange_charge; exchange_rate; exchange_via_app; extra_charge_on_statement; failed_transfer; fiat_currency_support; get_disposable_virtual_card; get_physical_card; getting_spare_card; getting_virtual_card; lost_or_stolen_card; lost_or_stolen_phone; order_physical_card; passcode_forgotten; pending_card_payment; pending_cash_withdrawal; pending_top_up; pending_transfer; pin_blocked; receiving_money; Refund_not_showing_up; request_refund; reverted_card_payment?; supported_cards_and_currencies; terminate_account; top_up_by_bank_transfer_charge; top_up_by_card_charge; top_up_by_cash_or_cheque; top_up_failed; top_up_limits; top_up_reverted; topping_up_by_card; transaction_charged_twice; transfer_fee_charged; transfer_into_account; transfer_not_received_by_recipient; transfer_timing; unable_to_verify_identity; verify_my_identity; verify_source_of_funds; verify_top_up; virtual_card_not_working; visa_or_mastercard; why_verify_identity; wrong_amount_of_cash_received; wrong_exchange_rate_for_cash_withdrawal\n",+ "\n",+ "[[ ## completed ## ]]\n",+ "\n",+ "In adhering to this structure, your objective is: \n",+ " Given the fields `text`, produce the fields `label`.\n",+ "\n",+ "\n",+ "\u001b[31mUser message:\u001b[0m\n",+ "\n",+ "[[ ## text ## ]]\n",+ "why hasnt my card come in yet?\n",+ "\n",+ "Respond with the corresponding output fields, starting with the field `[[ ## reasoning ## ]]`, then `[[ ## label ## ]]` (must be formatted as a valid Python Literal[activate_my_card, age_limit, apple_pay_or_google_pay, atm_support, automatic_top_up, balance_not_updated_after_bank_transfer, balance_not_updated_after_cheque_or_cash_deposit, beneficiary_not_allowed, cancel_transfer, card_about_to_expire, card_acceptance, card_arrival, card_delivery_estimate, card_linking, card_not_working, card_payment_fee_charged, card_payment_not_recognised, card_payment_wrong_exchange_rate, card_swallowed, cash_withdrawal_charge, cash_withdrawal_not_recognised, change_pin, compromised_card, contactless_not_working, country_support, declined_card_payment, declined_cash_withdrawal, declined_transfer, direct_debit_payment_not_recognised, disposable_card_limits, edit_personal_details, exchange_charge, exchange_rate, exchange_via_app, extra_charge_on_statement, failed_transfer, fiat_currency_support, get_disposable_virtual_card, get_physical_card, getting_spare_card, getting_virtual_card, lost_or_stolen_card, lost_or_stolen_phone, order_physical_card, passcode_forgotten, pending_card_payment, pending_cash_withdrawal, pending_top_up, pending_transfer, pin_blocked, receiving_money, Refund_not_showing_up, request_refund, reverted_card_payment?, supported_cards_and_currencies, terminate_account, top_up_by_bank_transfer_charge, top_up_by_card_charge, top_up_by_cash_or_cheque, top_up_failed, top_up_limits, top_up_reverted, topping_up_by_card, transaction_charged_twice, transfer_fee_charged, transfer_into_account, transfer_not_received_by_recipient, transfer_timing, unable_to_verify_identity, verify_my_identity, verify_source_of_funds, verify_top_up, virtual_card_not_working, visa_or_mastercard, why_verify_identity, wrong_amount_of_cash_received, wrong_exchange_rate_for_cash_withdrawal]), and then ending with the marker for `[[ ## completed ## ]]`.\n",+ "\n",+ "\n",+ "\u001b[31mResponse:\u001b[0m\n",+ "\n",+ "\u001b[32m[[ ## reasoning ## ]]\n",+ "The user is inquiring about the status of their card delivery, which suggests they are concerned about when they will receive their card. This aligns with the topic of card arrival and delivery estimates.\n",+ "\n",+ "[[ ## label ## ]]\n",+ "card_arrival\n",+ "\n",+ "[[ ## completed ## ]]\u001b[0m\n",+ "\n",+ "\n",+ "\n",+ "\n",+ "\n"+ ]+ }+ ],+ "source": [+ "classify_ft(text=\"why hasnt my card come in yet?\")\n",+ "dspy.inspect_history()"+ ]+ }+ ],+ "metadata": {+ "kernelspec": {+ "display_name": "py311_202501",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.11.9"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 2+}diff --git a/docs/docs/tutorials/index.md b/docs/docs/tutorials/index.mdindex 54cc02fb6..274cb88cd 100644--- a/docs/docs/tutorials/index.md+++ b/docs/docs/tutorials/index.md@@ -8,6 +8,8 @@* [Classification](/tutorials/classification/)+* [Classification Finetuning](/tutorials/classification_finetuning/)+* [Multi-Hop Search](/tutorials/multihop_search/)* [Privacy-Conscious Delegation](/tutorials/papillon/)diff --git a/docs/mkdocs.yml b/docs/mkdocs.ymlindex f30482d7a..4ad6f3060 100644--- a/docs/mkdocs.yml+++ b/docs/mkdocs.yml@@ -50,6 +50,7 @@ nav:- Reasoning: tutorials/math/index.ipynb- Entity Extraction: tutorials/entity_extraction/index.ipynb- Classification: tutorials/classification/index.md+ - Classification Finetuning: tutorials/classification_finetuning/index.ipynb- Multi-Hop Search: tutorials/multihop_search/index.ipynb- Privacy-Conscious Delegation: tutorials/papillon/index.md- Finetuning Agents: tutorials/games/index.ipynbdiff --git a/dspy/clients/lm.py b/dspy/clients/lm.pyindex 931540fa7..3639c4a25 100644--- a/dspy/clients/lm.py+++ b/dspy/clients/lm.py@@ -133,11 +133,11 @@ class LM(BaseLM):def launch(self, launch_kwargs: Optional[Dict[str, Any]] = None):launch_kwargs = launch_kwargs or self.launch_kwargs- self.provider.launch(self.model, launch_kwargs)+ self.provider.launch(self, launch_kwargs)def kill(self, launch_kwargs: Optional[Dict[str, Any]] = None):launch_kwargs = launch_kwargs or self.launch_kwargs- self.provider.kill(self.model, launch_kwargs)+ self.provider.kill(self, launch_kwargs)def finetune(self,diff --git a/dspy/clients/lm_local.py b/dspy/clients/lm_local.pynew file mode 100644index 000000000..c494a0fec--- /dev/null+++ b/dspy/clients/lm_local.py@@ -0,0 +1,330 @@+import time+import socket+import requests+import threading+import subprocess++from datasets import Dataset+from typing import Any, Dict, List, Optional+from dspy.clients.provider import TrainingJob, Provider+from dspy.clients.utils_finetune import DataFormat, TrainingStatus, save_data+++class LocalTrainingJob(TrainingJob):+ def __init__(self, *args, **kwargs):+ super().__init__(*args, **kwargs)+ self.data_path = None++ def cancel(self):+ raise NotImplementedError++ def status(self) -> TrainingStatus:+ raise NotImplementedError+++class LocalProvider(Provider):+ def __init__(self):+ super().__init__()+ self.finetunable = True+ self.TrainingJob = LocalTrainingJob++ @staticmethod+ def launch(lm: "LM", launch_kwargs: Optional[Dict[str, Any]] = None):+ try:+ import sglang+ except ImportError:+ raise ImportError(+ "For local model launching, please install sglang by running "+ '`pip install "sglang[all]"; pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/`'+ )++ if launch_kwargs is None:+ launch_kwargs = {}++ import os++ model = lm.model+ if model.startswith("openai/"):+ model = model[7:]+ if model.startswith("local:"):+ model = model[6:]++ print(f"Grabbing a free port to launch an SGLang server for model {model}")+ print(+ f"We see that CUDA_VISIBLE_DEVICES is {os.environ.get('CUDA_VISIBLE_DEVICES', 'unset')}"+ )+ port = get_free_port()+ timeout = launch_kwargs.get("timeout", 1800)+ command = f"python -m sglang.launch_server --model-path {model} --port {port} --host 0.0.0.0"++ # We will manually stream & capture logs.+ process = subprocess.Popen(+ command.replace("\\\n", " ").replace("\\", " ").split(),+ text=True,+ stdout=subprocess.PIPE, # We'll read from pipe+ stderr=subprocess.STDOUT, # Merge stderr into stdout+ )++ # A threading.Event to control printing after the server is ready.+ # This will store *all* lines (both before and after readiness).+ print(f"SGLang server process started with PID {process.pid}.")+ stop_printing_event = threading.Event()+ logs_buffer = []++ def _tail_process(proc, buffer, stop_event):+ while True:+ line = proc.stdout.readline()+ if not line and proc.poll() is not None:+ # Process ended and no new line+ break+ if line:+ buffer.append(line)+ # Print only if stop_event is not set+ if not stop_event.is_set():+ print(line, end="")++ # Start a background thread to read from the process continuously+ thread = threading.Thread(+ target=_tail_process,+ args=(process, logs_buffer, stop_printing_event),+ daemon=True,+ )+ thread.start()++ # Wait until the server is ready (or times out)+ base_url = f"http://localhost:{port}"+ try:+ wait_for_server(base_url, timeout=timeout)+ except TimeoutError:+ # If the server doesn't come up, we might want to kill it:+ process.kill()+ raise++ # Once server is ready, we tell the thread to stop printing further lines.+ stop_printing_event.set()++ # A convenience getter so the caller can see all logs so far (and future).+ def get_logs() -> str:+ # Join them all into a single string, or you might return a list+ return "".join(logs_buffer)++ # Let the user know server is up+ print(+ f"Server ready on random port {port}! Logs are available via lm.get_logs() method on returned lm."+ )++ lm.kwargs["api_base"] = f"http://localhost:{port}/v1"+ lm.kwargs["api_key"] = "local"+ lm.process = process+ lm.get_logs = get_logs++ @staticmethod+ def kill(model, kill_kwargs: Optional[Dict[str, Any]] = None):+ from sglang.utils import terminate_process++ terminate_process(model.process)++ @staticmethod+ def finetune(+ job: LocalTrainingJob,+ model: str,+ train_data: List[Dict[str, Any]],+ train_kwargs: Optional[Dict[str, Any]] = None,+ data_format: Optional[DataFormat] = None,+ ) -> str:+ data_path = save_data(train_data)++ if model.startswith("openai/"):+ model = model[7:]+ if model.startswith("local:"):+ model = model[6:]++ model_path = data_path.replace(".jsonl", f"__{model.replace('/', '__')}")+ print(f"[Local Provider] Data saved to {data_path}")++ default_train_kwargs = {+ "device": None,+ "use_peft": False,+ "num_train_epochs": 5,+ "per_device_train_batch_size": 1,+ "gradient_accumulation_steps": 8,+ "learning_rate": 1e-5,+ "max_seq_length": None,+ "packing": True,+ "bf16": True,+ "output_dir": model_path,+ }++ print(f"[Local Provider] Starting local training, will save to {model_path}")+ train_sft_locally(+ model_name=model,+ train_data=train_data,+ train_kwargs={**default_train_kwargs, **(train_kwargs or {})},+ )++ print("[Local Provider] Training complete")+ return f"openai/local:{model_path}"+++def train_sft_locally(model_name, train_data, train_kwargs):+ try:+ import torch+ from transformers import AutoModelForCausalLM, AutoTokenizer+ from trl import SFTConfig, SFTTrainer, setup_chat_format+ from trl import apply_chat_template+ except ImportError:+ raise ImportError(+ "For local finetuning, please install torch, transformers, and trl "+ "by running `pip install -U torch transformers accelerate trl peft`"+ )++ device = train_kwargs.get("device", None)+ if device is None:+ device = (+ "cuda"+ if torch.cuda.is_available()+ else "mps" if torch.backends.mps.is_available() else "cpu"+ )+ print(f"Using device: {device}")++ model = AutoModelForCausalLM.from_pretrained(+ pretrained_model_name_or_path=model_name+ ).to(device)+ tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path=model_name)++ # Set up the chat format; generally only for non-chat model variants, hence the try-except.+ try:+ model, tokenizer = setup_chat_format(model=model, tokenizer=tokenizer)+ except Exception:+ pass++ if tokenizer.pad_token_id is None:+ print("Adding pad token to tokenizer")+ tokenizer.add_special_tokens({"pad_token": "[!#PAD#!]"})++ print("Creating dataset")+ trainset_dict = {+ "prompt": [entry["messages"][:-1] for entry in train_data],+ "completion": [[entry["messages"][-1]] for entry in train_data],+ }++ trainset = Dataset.from_dict(trainset_dict)+ trainset = trainset.map(apply_chat_template, fn_kwargs={"tokenizer": tokenizer})++ USE_PEFT = train_kwargs.get("use_peft", False)+ peft_config = None++ if USE_PEFT:+ from peft import LoraConfig++ rank_dimension = 32+ lora_alpha = 64+ lora_dropout = 0.05++ peft_config = LoraConfig(+ r=rank_dimension,+ lora_alpha=lora_alpha,+ lora_dropout=lora_dropout,+ bias="none",+ target_modules="all-linear",+ task_type="CAUSAL_LM",+ )++ sft_config = SFTConfig(+ output_dir=output_dir,+ num_train_epochs=train_kwargs["num_train_epochs"],+ per_device_train_batch_size=train_kwargs["per_device_train_batch_size"],+ gradient_accumulation_steps=train_kwargs["gradient_accumulation_steps"],+ learning_rate=train_kwargs["learning_rate"],+ max_grad_norm=2.0,+ logging_steps=20,+ warmup_ratio=0.03,+ lr_scheduler_type="constant",+ save_steps=10_000,+ bf16=train_kwargs["bf16"],+ max_seq_length=train_kwargs["max_seq_length"],+ packing=train_kwargs["packing"],+ dataset_kwargs={+ "add_special_tokens": False, # Special tokens handled by template+ "append_concat_token": False, # No additional separator needed+ },+ )++ print("Starting training")++ trainer = SFTTrainer(+ model=model,+ args=sft_config,+ train_dataset=trainset,+ peft_config=peft_config,+ processing_class=tokenizer,+ )++ # Train!+ trainer.train()++ # Save the model!+ trainer.save_model()++ MERGE = True+ if USE_PEFT and MERGE:+ from peft import AutoPeftModelForCausalLM++ # Load PEFT model on CPU+ model_ = AutoPeftModelForCausalLM.from_pretrained(+ pretrained_model_name_or_path=sft_config.output_dir,+ torch_dtype=torch.float16,+ low_cpu_mem_usage=True,+ )++ merged_model = model_.merge_and_unload()+ merged_model.save_pretrained(+ sft_config.output_dir, safe_serialization=True, max_shard_size="5GB"+ )++ # Clean up!+ import gc++ del model+ del tokenizer+ del trainer+ gc.collect()+ torch.cuda.empty_cache()++ return output_dir+++def get_free_port() -> int:+ """+ Return a free TCP port on localhost.+ """+ with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:+ s.bind(("localhost", 0))+ return s.getsockname()[1]+++def wait_for_server(base_url: str, timeout: int = None) -> None:+ """+ Wait for the server to be ready by polling the /v1/models endpoint.++ Args:+ base_url: The base URL of the server (e.g. http://localhost:1234)+ timeout: Maximum time to wait in seconds. None means wait forever.+ """+ start_time = time.time()+ while True:+ try:+ response = requests.get(+ f"{base_url}/v1/models",+ headers={"Authorization": "Bearer None"},+ )+ if response.status_code == 200:+ # A small extra sleep to ensure server is fully up.+ time.sleep(5)+ break++ if timeout and (time.time() - start_time) > timeout:+ raise TimeoutError("Server did not become ready within timeout period")+ except requests.exceptions.RequestException:+ # Server not up yet, wait and retry+ time.sleep(1)diff --git a/dspy/teleprompt/bootstrap_finetune.py b/dspy/teleprompt/bootstrap_finetune.pyindex c69694625..d0d0bd7c0 100644--- a/dspy/teleprompt/bootstrap_finetune.py+++ b/dspy/teleprompt/bootstrap_finetune.py@@ -3,7 +3,7 @@ from typing import Any, Callable, Dict, List, Optional, Unionimport dspyfrom dspy.adapters.base import Adapter-from dspy.clients.lm import LM # TODO: Remove after the old LM class is removed+from dspy.clients.lm import LMfrom dspy.clients.utils_finetune import infer_data_formatfrom dspy.evaluate.evaluate import Evaluatefrom dspy.predict.predict import Predict@@ -30,8 +30,6 @@ class FinetuneTeleprompter(Teleprompter):class BootstrapFinetune(FinetuneTeleprompter):-- # TODO(PR) check with teamdef __init__(self,metric: Optional[Callable] = None,@@ -47,10 +45,6 @@ class BootstrapFinetune(FinetuneTeleprompter):# attached to LMs themselves -- an LM could know which adapter it should# be used with along with the train_kwargs. This will lead the only# required argument for LM.finetune() to be the train dataset.- err = "This is an experimental optimizer."- err += " Set `dspy.settings.experimental` to `True` to use it."- err += " Constructor arguments subject to change."- assert dspy.settings.experimental, errsuper().__init__(train_kwargs=train_kwargs)self.metric = metric@@ -146,6 +140,9 @@ class BootstrapFinetune(FinetuneTeleprompter):call_data = build_call_data_from_trace(trace=item['trace'], pred_ind=pred_ind, adapter=adapter, exclude_demos=self.exclude_demos)data.append(call_data)+ import random+ random.Random(0).shuffle(data)+return data, data_format