DSPy: The framework for programming—not prompting—language models
Python
Add two initial dspy.GRPO tutorials (#8172)
Omar Khattab983e7921f2c8
diff --git a/docs/docs/tutorials/rl_multihop/index.ipynb b/docs/docs/tutorials/rl_multihop/index.ipynbnew file mode 100644index 000000000..9f1ab2e11--- /dev/null+++ b/docs/docs/tutorials/rl_multihop/index.ipynb@@ -0,0 +1,338 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "# Tutorial: Online RL for Multi-Hop Research\n",+ "\n",+ "WARNING: This feature is new and extremely EXPERIMENTAL. Unlike almost everything else in DSPy, it's currently in pure proof of concept and development mode, but we release it to encourage community involvement.\n",+ "\n",+ "If you want to be on the cutting edge even before it's merged, install the `dspy.GRPO` PR via `pip install git+https://github.com/stanfordnlp/dspy.git@refs/pull/8171/head` and follow along.\n",+ "\n",+ "For this tutorial, you will also need DSPy's Arbor RL server.\n",+ "\n",+ "```bash\n",+ "> pip install git+https://github.com/Ziems/arbor.git\n",+ "> python -m arbor.cli serve --arbor-config arbor.yaml\n",+ "```\n",+ "\n",+ "where you create `arbor.yaml` in your directory, containing a plan like:\n",+ "\n",+ "```text\n",+ "inference:\n",+ " gpu_ids: '0,'\n",+ "\n",+ "training:\n",+ " gpu_ids: '1, 2'\n",+ "```\n",+ "\n",+ "which assigns GPU 0 for inference and GPUs 1 and 2 for training."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "import dspy\n",+ "from dspy.clients.lm_local_arbor import ArborProvider\n",+ "\n",+ "port = 7453\n",+ "local_lm_name = \"Qwen/Qwen3-1.7B\"\n",+ "local_lm = dspy.LM(\n",+ " model=f\"openai/arbor:{local_lm_name}\",\n",+ " provider=ArborProvider(),\n",+ " temperature=0.7,\n",+ " api_base=f\"http://localhost:{port}/v1/\",\n",+ " api_key=\"arbor\",\n",+ ")\n",+ "\n",+ "dspy.configure(lm=local_lm)\n",+ "\n",+ "openai_lm = dspy.LM(model=\"openai/gpt-4.1-mini\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Install dependencies and download data\n",+ "\n",+ "To do the retrieval, we'll use the cool BM25S library, as it's pretty lightweight. You can replace this components with whatever you like.\n",+ "\n",+ "```shell\n",+ "> pip install -U bm25s PyStemmer \"jax[cpu]\"\n",+ "```\n",+ "\n",+ "Next, we'll download a snapshot abstracts (i.e., first paragraphs) of all 5,000,000 Wikipedia pages as of 2017. We'll use this as our retrieval corpus.\n",+ "\n",+ "This is 500MB compressed, so the download and decompression may take 2-3 minutes.\n",+ "\n",+ "```python\n",+ "from dspy.utils import download\n",+ "\n",+ "download(\"https://huggingface.co/dspy/cache/resolve/main/wiki.abstracts.2017.tar.gz\")\n",+ "!tar -xzvf wiki.abstracts.2017.tar.gz\n",+ "```\n",+ "\n",+ "And then let's index it for BM25 retrieval! This will take 2-3 minutes."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "import ujson\n",+ "import bm25s\n",+ "import Stemmer\n",+ "\n",+ "corpus = []\n",+ "\n",+ "with open(\"wiki.abstracts.2017.jsonl\") as f:\n",+ " for line in f:\n",+ " line = ujson.loads(line)\n",+ " corpus.append(f\"{line['title']} | {' '.join(line['text'])}\")\n",+ "\n",+ "stemmer = Stemmer.Stemmer(\"english\")\n",+ "corpus_tokens = bm25s.tokenize(corpus, stopwords=\"en\", stemmer=stemmer)\n",+ "\n",+ "retriever = bm25s.BM25(k1=0.9, b=0.4)\n",+ "retriever.index(corpus_tokens)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Load the HoVer dataset.\n",+ "\n",+ "Let's load a dataset for our task. We'll load examples from the HoVer multi-hop task, where the input is a (really!) complex claim and the output we're seeking is the set of Wikipedia pages that are required to fact-check that claim."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "import random\n",+ "from dspy.datasets import DataLoader\n",+ "\n",+ "kwargs = dict(fields=(\"claim\", \"supporting_facts\", \"hpqa_id\", \"num_hops\"), input_keys=(\"claim\",))\n",+ "hover = DataLoader().from_huggingface(dataset_name=\"hover-nlp/hover\", split=\"train\", trust_remote_code=True, **kwargs)\n",+ "\n",+ "hpqa_ids = set()\n",+ "hover = [\n",+ " dspy.Example(claim=x.claim, titles=list(set([y[\"key\"] for y in x.supporting_facts]))).with_inputs(\"claim\")\n",+ " for x in hover\n",+ " if x[\"num_hops\"] == 3 and x[\"hpqa_id\"] not in hpqa_ids and not hpqa_ids.add(x[\"hpqa_id\"])\n",+ "]\n",+ "\n",+ "random.Random(0).shuffle(hover)\n",+ "trainset, devset, testset = hover[:600], hover[600:900], hover[900:]\n",+ "len(trainset), len(devset), len(testset)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Now, let's define a function to do the search in Wikipedia. This will use our BM25 index."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "def search(query: str, k: int) -> list[str]:\n",+ " tokens = bm25s.tokenize(query, stopwords=\"en\", stemmer=stemmer, show_progress=False)\n",+ " results, scores = retriever.retrieve(tokens, k=k, n_threads=1, show_progress=False)\n",+ " run = {corpus[doc]: float(score) for doc, score in zip(results[0], scores[0])}\n",+ " return list(run.keys())"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## A DSPy program for multi-hop research\n",+ "\n",+ "Now, let's define the multi-hop program in DSPy. It's going to be super simple, composed of `generate_query` and `append_notes` modules. We'll define the instructions carefully, though they are typically not necessary."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "instr1 = \"\"\"\n",+ "Given a claim and some key facts, generate a follow-up search query to find the next most essential clue towards verifying or refuting the claim. The goal ultimately is to find all documents implicated by the claim.\n",+ "\"\"\".strip()\n",+ "\n",+ "instr2 = \"\"\"\n",+ "Given a claim, some key facts, and new search results, identify any new learnings from the new search results, which will extend the key facts known so far about the whether the claim is true or false. The goal is to ultimately collect all facts that would help us find all documents implicated by the claim.\n",+ "\"\"\"\n",+ "\n",+ "\n",+ "class ResearchHop(dspy.Module):\n",+ " def __init__(self, num_docs, num_hops):\n",+ " self.num_docs, self.num_hops = num_docs, num_hops\n",+ " self.generate_query = dspy.ChainOfThought(dspy.Signature(\"claim, key_facts -> followup_search_query\", instr1))\n",+ " self.append_notes = dspy.ChainOfThought(dspy.Signature(\"claim, key_facts, new_search_results -> new_key_facts\", instr2))\n",+ "\n",+ " def forward(self, claim: str) -> list[str]:\n",+ " key_facts = []\n",+ " retrieved_docs = []\n",+ "\n",+ " for hop_idx in range(self.num_hops):\n",+ " query = self.generate_query(claim=claim, key_facts=key_facts).followup_search_query if hop_idx else claim\n",+ " search_results = search(query, k=self.num_docs)\n",+ " retrieved_docs.extend(search_results)\n",+ "\n",+ " if hop_idx == self.num_hops - 1:\n",+ " break\n",+ " \n",+ " prediction = self.append_notes(claim=claim, key_facts=key_facts, new_search_results=search_results)\n",+ " key_facts.append(prediction.new_key_facts)\n",+ "\n",+ " return dspy.Prediction(key_facts=key_facts, retrieved_docs=retrieved_docs)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Define metrics for success in this task"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "def recall(example, pred, trace=None):\n",+ " gold_titles = example.titles\n",+ " retrieved_titles = [doc.split(\" | \")[0] for doc in pred.retrieved_docs]\n",+ " return sum(x in retrieved_titles for x in set(gold_titles)) / len(gold_titles)\n",+ "\n",+ "evaluate = dspy.Evaluate(devset=devset, metric=recall, num_threads=16, display_progress=True, display_table=5)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## Optimize the `ResearchHop` system with `dspy.GRPO`"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "from dspy.teleprompt.grpo import GRPO\n",+ "\n",+ "program = ResearchHop(num_docs=4, num_hops=2)\n",+ "program.set_lm(local_lm)\n",+ "\n",+ "# NOTE: Training on 6 GPUs.\n",+ "train_kwargs = {\n",+ " \"update_interval\": 3,\n",+ " \"per_device_train_batch_size\": 2,\n",+ " \"gradient_accumulation_steps\": 4,\n",+ " \"temperature\": 0.7,\n",+ " \"beta\": 0.04,\n",+ " \"learning_rate\": 2e-5,\n",+ " \"gradient_checkpointing\": True,\n",+ " \"gradient_checkpointing_kwargs\": {\"use_reentrant\": False},\n",+ " \"bf16\": True,\n",+ " \"lr_scheduler_type\": \"constant_with_warmup\",\n",+ " \"max_prompt_length\": None,\n",+ " \"max_completion_length\": None,\n",+ " \"scale_rewards\": True,\n",+ " \"max_grad_norm\": 0.5,\n",+ " \"lora\": True,\n",+ "}\n",+ "\n",+ "compiler = GRPO(\n",+ " metric=recall,\n",+ " multitask=True,\n",+ " num_dspy_examples_per_grpo_step=6,\n",+ " num_samples_per_input=8,\n",+ " exclude_demos=True,\n",+ " num_train_steps=500,\n",+ " num_threads=24,\n",+ " use_train_as_val=False,\n",+ " num_steps_for_val=10,\n",+ " train_kwargs=train_kwargs,\n",+ " report_train_scores=False,\n",+ ")\n",+ "\n",+ "optimized_program = compiler.compile(\n",+ " student=program,\n",+ " trainset=trainset,\n",+ " valset=devset,\n",+ ")\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Now, you can use the GRPO'ed program."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "example = devset[0]\n",+ "optimized_program(**example.inputs())"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "In our preliminary experiments, training above for about 18 hours boosts the recall (devset) from 61.8% to 66.2%. This is _typically_ worse on cost/quality basis than you'd get from running prompt optimizers dspy.MIPROv2 or dspy.SIMBA, but it's still a very solid start for online RL over arbitrary LM programs for small LMs."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": []+ }+ ],+ "metadata": {+ "kernelspec": {+ "display_name": "jun2024_py310",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.10.14"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 2+}diff --git a/docs/docs/tutorials/rl_papillon/index.ipynb b/docs/docs/tutorials/rl_papillon/index.ipynbnew file mode 100644index 000000000..9939f897e--- /dev/null+++ b/docs/docs/tutorials/rl_papillon/index.ipynb@@ -0,0 +1,270 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "# Tutorial: Online RL over a Multi-Module DSPy Program\n",+ "\n",+ "WARNING: This feature is new and extremely EXPERIMENTAL. Unlike almost everything else in DSPy, it's currently in pure proof of concept and development mode, but we release it to encourage community involvement.\n",+ "\n",+ "Install the latest DSPy via `pip install -U dspy` and follow along.\n",+ "\n",+ "In this tutorial, we optimize the LM weights of [PAPILLON](https://dspy.ai/tutorials/papillon/) with `dspy.GRPO`, a generalization of the popular GRPO online RL algorithm of LLMs to sophisticated multi-module LM programs.\n",+ "\n",+ "PAPILLON is a system for privacy-preserving delegation, where we will teach a tiny model (1.7B parameters) to use an \"untrusted\" external LLM, which is more powerful but may save your private data, to balance high-quality and private chat.\n",+ "\n",+ "For this tutorial, you will also need the Arbor RL server.\n",+ "\n",+ "```bash\n",+ "> pip install git+https://github.com/Ziems/arbor.git\n",+ "> python -m arbor.cli serve --arbor-config arbor.yaml\n",+ "```\n",+ "\n",+ "where you create `arbor.yaml` in your directory, containing a plan like:\n",+ "\n",+ "```text\n",+ "inference:\n",+ " gpu_ids: '0,'\n",+ "\n",+ "training:\n",+ " gpu_ids: '1, 2'\n",+ "```\n",+ "\n",+ "which assigns GPU 0 for inference and GPUs 1 and 2 for training."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "import dspy\n",+ "from dspy.clients.lm_local_arbor import ArborProvider\n",+ "\n",+ "port = 7453\n",+ "local_lm_name = \"Qwen/Qwen3-1.7B\"\n",+ "local_lm = dspy.LM(\n",+ " model=f\"openai/arbor:{local_lm_name}\",\n",+ " provider=ArborProvider(),\n",+ " temperature=0.7,\n",+ " api_base=f\"http://localhost:{port}/v1/\",\n",+ " api_key=\"arbor\",\n",+ ")\n",+ "\n",+ "dspy.configure(lm=local_lm)\n",+ "\n",+ "openai_lm = dspy.LM(model=\"openai/gpt-4.1-mini\")"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "class CraftRedactedRequest(dspy.Signature):\n",+ " \"\"\"\n",+ " Given a private user query, create a privacy-preserving request for a powerful external LLM.\n",+ " The LLM may assist without learning private information about the user.\n",+ " \"\"\"\n",+ "\n",+ " user_query = dspy.InputField()\n",+ " llm_request = dspy.OutputField()\n",+ "\n",+ "\n",+ "class RespondToQuery(dspy.Signature):\n",+ " \"\"\"\n",+ " Respond to a user query.\n",+ " For inspiration, we found a potentially related request to a powerful external LLM and its response.\n",+ " \"\"\"\n",+ "\n",+ " related_llm_request = dspy.InputField()\n",+ " related_llm_response = dspy.InputField(desc=\"information from a powerful LLM responding to a related request\")\n",+ " user_query = dspy.InputField(desc=\"the user's request you need to fulfill\")\n",+ " response = dspy.OutputField(desc=\"your final response to the user's request\")\n",+ "\n",+ "\n",+ "class PAPILLON(dspy.Module):\n",+ " def __init__(self, untrusted_model):\n",+ " self.craft_redacted_request = dspy.ChainOfThought(CraftRedactedRequest)\n",+ " self.respond_to_query = dspy.Predict(RespondToQuery)\n",+ " self.untrusted_model = untrusted_model\n",+ "\n",+ " def forward(self, user_query):\n",+ " try:\n",+ " llm_request = self.craft_redacted_request(user_query=user_query).llm_request\n",+ " llm_response = self.untrusted_model(llm_request)[0]\n",+ " response = self.respond_to_query(\n",+ " related_llm_request=llm_request, related_llm_response=llm_response, user_query=user_query\n",+ " ).response\n",+ " except Exception:\n",+ " return dspy.Prediction(llm_request=\"\", llm_response=\"\", response=\"\")\n",+ "\n",+ " return dspy.Prediction(llm_request=llm_request, llm_response=llm_response, response=response)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "from datasets import load_dataset\n",+ "\n",+ "pupa_tnb = load_dataset(\"Columbia-NLP/PUPA\", \"pupa_tnb\")\n",+ "pupa_new = load_dataset(\"Columbia-NLP/PUPA\", \"pupa_new\")\n",+ "\n",+ "examples = [\n",+ " dspy.Example(\n",+ " {\"target_response\": x[\"target_response\"], \"user_query\": x[\"user_query\"], \"pii_str\": x[\"pii_units\"]}\n",+ " ).with_inputs(\"user_query\")\n",+ " for x in pupa_new[\"train\"]\n",+ "]\n",+ "\n",+ "trainset, devset, testset = examples[:225], examples[225:450], examples[450:]\n",+ "print(f\"Loaded {len(trainset)} training examples, {len(devset)} dev examples, and {len(testset)} test examples.\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Define metrics for success in this task\n",+ "\n",+ "What does it mean for a PAPILLON system to be successful?\n",+ "\n",+ "1. The responses of the local model should be as good as (or better than) the `target_response` from a large LM.\n",+ "2. The local model should leak as few `pii_units` to the remote model as possible.\n",+ "\n",+ "For benchmarking, we will judge both of these using our `openai_lm` and the annotation in PUPA."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "zeroshot = PAPILLON(untrusted_model=openai_lm)\n",+ "\n",+ "kwargs = dict(num_threads=16, display_progress=True, display_table=5, max_errors=100)\n",+ "evaluate = dspy.Evaluate(metric=compute_overall_score, devset=devset, **kwargs)\n",+ "evaluate(zeroshot)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## Optimize PAPILLON with `dspy.GRPO`\n",+ "\n",+ "Let's run the `dspy.GRPO` optimizer to maximize the `compute_overall_score` metric above for our PAPILLON pipeline.\n",+ "\n",+ "We ran this on 4xH100 GPUs for a couple of hours. But first, you'll need to set up Arbor (as above)."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "from dspy.teleprompt.grpo import GRPO\n",+ "\n",+ "papillon = PAPILLON(untrusted_model=openai_lm)\n",+ "papillon.set_lm(local_lm)\n",+ "\n",+ "# NOTE: Training on 3 GPUs.\n",+ "train_kwargs = {\n",+ " \"update_interval\": 3,\n",+ " \"per_device_train_batch_size\": 8,\n",+ " \"gradient_accumulation_steps\": 4,\n",+ " \"temperature\": 0.7,\n",+ " \"beta\": 0.04,\n",+ " \"learning_rate\": 2e-6,\n",+ " \"gradient_checkpointing\": True,\n",+ " \"gradient_checkpointing_kwargs\": {\"use_reentrant\": False},\n",+ " \"bf16\": True,\n",+ " \"lr_scheduler_type\": \"constant_with_warmup\",\n",+ " \"max_prompt_length\": None,\n",+ " \"max_completion_length\": None,\n",+ " \"scale_rewards\": True,\n",+ " \"max_grad_norm\": 0.5,\n",+ " \"lora\": True,\n",+ "}\n",+ "\n",+ "compiler = GRPO(\n",+ " metric=compute_overall_score,\n",+ " multitask=True,\n",+ " num_dspy_examples_per_grpo_step=4,\n",+ " num_samples_per_input=8,\n",+ " exclude_demos=True,\n",+ " num_train_steps=500,\n",+ " num_threads=24,\n",+ " use_train_as_val=False,\n",+ " num_steps_for_val=10,\n",+ " train_kwargs=train_kwargs,\n",+ " report_train_scores=False,\n",+ ")\n",+ "\n",+ "optimized_papillon = compiler.compile(\n",+ " student=papillon,\n",+ " trainset=trainset,\n",+ " valset=devset,\n",+ ")\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Now, you can use the GRPO'ed program."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "example = devset[0]\n",+ "optimized_papillon(**example.inputs())"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "In our preliminary experiments, training above for three hours boosts the composite score (devset) from 54.6% to 60.0%. This is _typically_ worse on cost/quality basis than you'd get from running prompt optimizers like dspy.MIPROv2 or dspy.SIMBA, but it's still a very solid start for online RL over arbitrary LM programs for tiny LMs."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": []+ }+ ],+ "metadata": {+ "kernelspec": {+ "display_name": "jun2024_py310",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.10.14"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 2+}