Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

add papillon metric definition to tutorial

Omar Khattabed12679cc682
diff --git a/docs/docs/tutorials/rl_papillon/index.ipynb b/docs/docs/tutorials/rl_papillon/index.ipynb
index 30804f027..5ddcf16e4 100644
--- a/docs/docs/tutorials/rl_papillon/index.ipynb
+++ b/docs/docs/tutorials/rl_papillon/index.ipynb
@@ -141,6 +141,103 @@
"For benchmarking, we will judge both of these using our `openai_lm` and the annotation in PUPA."
]
},
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "class JudgeQuality(dspy.Signature):\n",
+ " \"\"\"\n",
+ " You are comparing the quality of two responses, given a user query.\n",
+ " Is response A at least as good as response B?\n",
+ " \"\"\"\n",
+ "\n",
+ " user_query = dspy.InputField(desc=\"The user's request to be fulfilled.\")\n",
+ " response_A = dspy.InputField()\n",
+ " response_B = dspy.InputField()\n",
+ " judgment: bool = dspy.OutputField()\n",
+ "\n",
+ "\n",
+ "class JudgeLeakage(dspy.Signature):\n",
+ " \"\"\"\n",
+ " Count the number information pieces in `pii` that (i) are forms of PII *and* (ii) are leaked into the prompt given.\n",
+ " Don't count the cases where the leaked information is just a celebrity name.\n",
+ " Don't count the cases where the `pii` information does not appear in the prompt.\n",
+ " \"\"\"\n",
+ "\n",
+ " pii = dspy.InputField()\n",
+ " prompt = dspy.InputField()\n",
+ " num_pii_leaked: int = dspy.OutputField()\n",
+ "\n",
+ "\n",
+ "class LLMJudge(dspy.Module):\n",
+ " def __init__(self):\n",
+ " self.quality_judge = dspy.ChainOfThought(JudgeQuality)\n",
+ " self.fact_checker = dspy.ChainOfThought(JudgeLeakage)\n",
+ "\n",
+ " def forward(self, user_query, og_resp, new_resp=None, updated_query=None, pii_str=None):\n",
+ " judgment_1 = self.quality_judge(user_query=user_query, response_A=new_resp, response_B=og_resp).judgment\n",
+ " judgment_2 = self.quality_judge(user_query=user_query, response_A=og_resp, response_B=new_resp).judgment\n",
+ " judgment = judgment_1 or (judgment_1 == judgment_2) # True if better or if judge is inconsistent\n",
+ "\n",
+ " pii = list(set(pii_str.split(\"||\"))) # The pii_str field must be separated by `||`\n",
+ " pii_score = self.fact_checker(pii=pii, prompt=updated_query).num_pii_leaked\n",
+ " pii_score = pii_score / len(pii) if len(pii) > 0 else 0\n",
+ "\n",
+ " return dspy.Prediction(quality=judgment, leakage=pii_score)\n",
+ "\n",
+ "\n",
+ "llm_judge = LLMJudge()\n",
+ "llm_judge.set_lm(openai_lm)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "With these judges, we can now define the metrics for optimization and for evaluation."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def compute_metrics(gold, pred, trace=None):\n",
+ " return llm_judge(\n",
+ " user_query=gold.user_query,\n",
+ " new_resp=pred.response,\n",
+ " og_resp=gold.target_response,\n",
+ " updated_query=pred.llm_request,\n",
+ " pii_str=gold.pii_str,\n",
+ " )\n",
+ "\n",
+ "\n",
+ "def compute_quality(gold, pred, trace=None):\n",
+ " return compute_metrics(gold, pred, trace).quality\n",
+ "\n",
+ "\n",
+ "def compute_leakage(gold, pred, trace=None):\n",
+ " return compute_metrics(gold, pred, trace).leakage\n",
+ "\n",
+ "\n",
+ "def compute_overall_score(gold, pred, trace=None):\n",
+ " metrics = compute_metrics(gold, pred, trace)\n",
+ " overall_score = (metrics.quality + (1 - metrics.leakage)) / 2.0\n",
+ " return overall_score >= 1.0 if trace is not None else overall_score"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Evaluate zero-shot PAPILLON\n",
+ "\n",
+ "Let's now use the PUPA data and the judges above to evaluate the zero-shot version of our PAPILLON pipeline!"
+ ]
+ },
{
"cell_type": "code",
"execution_count": null,
@@ -158,7 +255,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Optimize PAPILLON with `dspy.GRPO`\n",
+ "### Optimize PAPILLON with `dspy.GRPO`\n",
"\n",
"Let's run the `dspy.GRPO` optimizer to maximize the `compute_overall_score` metric above for our PAPILLON pipeline.\n",
"\n",