DSPy: The framework for programming—not prompting—language models
Python
add papillon metric definition to tutorial
Omar Khattabed12679cc682
diff --git a/docs/docs/tutorials/rl_papillon/index.ipynb b/docs/docs/tutorials/rl_papillon/index.ipynbindex 30804f027..5ddcf16e4 100644--- a/docs/docs/tutorials/rl_papillon/index.ipynb+++ b/docs/docs/tutorials/rl_papillon/index.ipynb@@ -141,6 +141,103 @@"For benchmarking, we will judge both of these using our `openai_lm` and the annotation in PUPA."]},+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "class JudgeQuality(dspy.Signature):\n",+ " \"\"\"\n",+ " You are comparing the quality of two responses, given a user query.\n",+ " Is response A at least as good as response B?\n",+ " \"\"\"\n",+ "\n",+ " user_query = dspy.InputField(desc=\"The user's request to be fulfilled.\")\n",+ " response_A = dspy.InputField()\n",+ " response_B = dspy.InputField()\n",+ " judgment: bool = dspy.OutputField()\n",+ "\n",+ "\n",+ "class JudgeLeakage(dspy.Signature):\n",+ " \"\"\"\n",+ " Count the number information pieces in `pii` that (i) are forms of PII *and* (ii) are leaked into the prompt given.\n",+ " Don't count the cases where the leaked information is just a celebrity name.\n",+ " Don't count the cases where the `pii` information does not appear in the prompt.\n",+ " \"\"\"\n",+ "\n",+ " pii = dspy.InputField()\n",+ " prompt = dspy.InputField()\n",+ " num_pii_leaked: int = dspy.OutputField()\n",+ "\n",+ "\n",+ "class LLMJudge(dspy.Module):\n",+ " def __init__(self):\n",+ " self.quality_judge = dspy.ChainOfThought(JudgeQuality)\n",+ " self.fact_checker = dspy.ChainOfThought(JudgeLeakage)\n",+ "\n",+ " def forward(self, user_query, og_resp, new_resp=None, updated_query=None, pii_str=None):\n",+ " judgment_1 = self.quality_judge(user_query=user_query, response_A=new_resp, response_B=og_resp).judgment\n",+ " judgment_2 = self.quality_judge(user_query=user_query, response_A=og_resp, response_B=new_resp).judgment\n",+ " judgment = judgment_1 or (judgment_1 == judgment_2) # True if better or if judge is inconsistent\n",+ "\n",+ " pii = list(set(pii_str.split(\"||\"))) # The pii_str field must be separated by `||`\n",+ " pii_score = self.fact_checker(pii=pii, prompt=updated_query).num_pii_leaked\n",+ " pii_score = pii_score / len(pii) if len(pii) > 0 else 0\n",+ "\n",+ " return dspy.Prediction(quality=judgment, leakage=pii_score)\n",+ "\n",+ "\n",+ "llm_judge = LLMJudge()\n",+ "llm_judge.set_lm(openai_lm)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "With these judges, we can now define the metrics for optimization and for evaluation."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "def compute_metrics(gold, pred, trace=None):\n",+ " return llm_judge(\n",+ " user_query=gold.user_query,\n",+ " new_resp=pred.response,\n",+ " og_resp=gold.target_response,\n",+ " updated_query=pred.llm_request,\n",+ " pii_str=gold.pii_str,\n",+ " )\n",+ "\n",+ "\n",+ "def compute_quality(gold, pred, trace=None):\n",+ " return compute_metrics(gold, pred, trace).quality\n",+ "\n",+ "\n",+ "def compute_leakage(gold, pred, trace=None):\n",+ " return compute_metrics(gold, pred, trace).leakage\n",+ "\n",+ "\n",+ "def compute_overall_score(gold, pred, trace=None):\n",+ " metrics = compute_metrics(gold, pred, trace)\n",+ " overall_score = (metrics.quality + (1 - metrics.leakage)) / 2.0\n",+ " return overall_score >= 1.0 if trace is not None else overall_score"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Evaluate zero-shot PAPILLON\n",+ "\n",+ "Let's now use the PUPA data and the judges above to evaluate the zero-shot version of our PAPILLON pipeline!"+ ]+ },{"cell_type": "code","execution_count": null,@@ -158,7 +255,7 @@"cell_type": "markdown","metadata": {},"source": [- "## Optimize PAPILLON with `dspy.GRPO`\n",+ "### Optimize PAPILLON with `dspy.GRPO`\n","\n","Let's run the `dspy.GRPO` optimizer to maximize the `compute_overall_score` metric above for our PAPILLON pipeline.\n","\n",