Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

chore(assistant): refresh embedded Langfuse skills (#17916)

langfuse-bot41d4ac3cb953
diff --git a/packages/langfuse-skills/src/generated/skills.js b/packages/langfuse-skills/src/generated/skills.js
index f080237a5d..10c055bf68 100644
--- a/packages/langfuse-skills/src/generated/skills.js
+++ b/packages/langfuse-skills/src/generated/skills.js
@@ -40,9 +40,9 @@ exports.LANGFUSE_SKILLS = [
{
name: "langfuse-judge-calibration",
description:
- "Calibrate and validate LLM-as-a-Judge evaluators against dataset ground truth. Runs the judge prompt as a Langfuse dataset experiment, compares judge outputs with dataset item expected outputs, and reports simple accuracy or advanced confusion-matrix metrics. Use this guide whenever a user asks if their LLM judge is actually useful, aligned with human judgment, or safe to trust for monitoring decisions.",
+ "Calibrate a new or existing LLM-as-a-Judge against labeled examples, analyze disagreements, iterate on its prompt with user approval, and deploy the approved version.",
instructions:
- '# Judge Calibration (LLM-as-a-Judge)\n\n## Goal\n\nValidate judge outputs against human labels using the smallest reliable workflow\nfor the user\'s goal.\n\nDefault to a **Langfuse dataset experiment** when the user has a Langfuse\ndataset or wants results in the Langfuse Experiments UI.\n\nDefault to **simple calibration** unless the user asks for deeper metrics,\nsplit-based validation, thresholding, or production automation.\n\n## 1) Choose the calibration mode\n\n### Simple calibration\n\nUse this when the user wants a quick answer like "does this judge basically\nmatch human labels?" or explicitly asks for accuracy only.\n\n- No train/dev/test split is required.\n- Compute `exact_match` for each valid row.\n- Report valid sample size, invalid-label count, accuracy, and a short\n recommendation.\n- Do not include Precision/Recall/F1, TPR/TNR, denominator notes, or top failure\n direction unless the user asks for advanced metrics.\n\n### Advanced calibration\n\nUse this when the user asks for confusion matrix metrics, thresholds,\nproduction monitoring, high-stakes automation, or train/test-style validation.\n\n- If split labels exist, keep train/dev/test separate.\n- If no split exists, compute metrics on the provided rows and state that this is\n not a held-out final quality claim.\n- Compute TP/FP/FN/TN and derived metrics.\n- Use `references/error-analysis.md` for qualitative diagnosis of disagreements.\n\n## 2) Primary workflow\n\n1. Confirm the dataset name, ground-truth label location in `expectedOutput`,\n judge prompt name/version, judge model, and label vocabulary.\n2. Choose simple or advanced mode. If ambiguous, use simple mode.\n3. Run the judge prompt against each dataset item input as a Langfuse experiment.\n4. Compare the judge output to `item.expected_output` in evaluator functions.\n5. Return the matching report format from section 7.\n\n## 3) Langfuse experiment workflow\n\nUse the SDK experiment runner as the default implementation. A Langfuse-hosted\ndataset automatically creates a dataset run that can be inspected and compared\nin the Langfuse UI.\n\nBefore implementing, you **must** retrieve the current experiment SDK\ndocumentation from the Langfuse docs — do not rely on memory, the SDK changes\nfrequently. Fetch these pages (see SKILL.md section 2 for retrieval methods):\n\n- [Experiments via SDK](https://langfuse.com/docs/evaluation/experiments/experiments-via-sdk) — primary reference for `dataset.run_experiment`, task/evaluator signatures, and `Evaluation` return shape\n- [Datasets](https://langfuse.com/docs/evaluation/experiments/datasets) — dataset item structure (`input`, `expectedOutput`) and how to load a hosted dataset\n- [Experiments data model](https://langfuse.com/docs/evaluation/experiments/data-model) — how runs, items, and scores relate in the UI\n\nHigh-level shape of a simple-mode calibration experiment:\n\n```\nload dataset and judge prompt from Langfuse\ndefine POSITIVE / NEGATIVE label set\n\ntask(item):\n compile judge prompt with item.input\n call judge model\n return normalized label\n # never read item.expected_output here — that would leak the answer\n\nitem_evaluator(output, expected_output):\n if either label is outside the allowed set:\n return invalid (excluded from accuracy denominator)\n return exact_match score (0 or 1)\n\nrun_evaluator(item_results):\n accuracy = matches / valid_rows\n return aggregate score + invalid_label count\n\ndataset.run_experiment(task, [item_evaluator], [run_evaluator],\n metadata={calibration_mode, judge_prompt, labels})\n```\n\nFor advanced calibration, the item evaluator emits one score per confusion-matrix\ncell (`judge-is-tp`, `judge-is-fp`, `judge-is-fn`, `judge-is-tn`), and the run\nevaluator aggregates them into Precision / Recall / F1 / TPR / TNR. See section\n6 for the metric definitions and zero-denominator guardrails.\n\nRules:\n- Use a Langfuse-hosted dataset when the user wants a real Langfuse experiment.\n Local SDK datasets create traces and scores, but not Langfuse dataset runs.\n- The dataset item `input` must contain everything needed to run the judge\n prompt. The dataset item `expectedOutput` must contain the ground-truth label.\n- Never pass `expectedOutput` into the judge prompt or task. That would leak the\n answer and invalidate calibration.\n- Return `Evaluation(...)` objects from item and run evaluators for stable SDK\n formatting and score ingestion.\n- Store prompt name/version, judge model, label vocabulary, dataset version, and\n calibration mode in run metadata.\n- Use a unique run name so the experiment appears as a separate dataset run.\n\n## 4) Label validation\n\nNever silently treat unknown labels as negative.\n\n- For binary judges, define the positive and negative labels before computing\n confusion-matrix metrics.\n- Normalize only deterministic differences such as surrounding whitespace or\n casing, and mention that normalization was applied.\n- Rows where `expected` or `actual` is outside the allowed label set are invalid.\n Exclude them from metric denominators and report the invalid count.\n- For multi-class judges, use simple `exact_match` / accuracy unless the user\n defines one positive class for binary metrics.\n\nExample binary labels:\n- Positive: `ESCALATE`\n- Negative: `RESOLVE`\n\n## 5) Simple metrics\n\nFor each valid row:\n\n- `exact_match = 1 if actual == expected else 0`\n\nAggregate:\n\n- `accuracy = sum(exact_match) / valid_rows`\n\nIf `valid_rows == 0`, report that accuracy is undefined and ask for valid\nexpected/actual labels.\n\n## 6) Advanced metrics\n\n### Dataset and split discipline\n\nUse split discipline only when the user asks for advanced validation or final\nquality claims.\n\n- **Train**: optional few-shot examples for the judge prompt\n- **Dev**: iterative prompt/model refinement\n- **Test**: single final calibration pass\n\nDo not tune on the same rows used to claim final quality. Use balanced classes\nin dev/test when possible so both error directions are measurable.\n\n### Per-row classification mapping\n\nFor each row with `expected` and `actual` labels:\n\n- **TP**: expected = positive and actual = positive\n- **FP**: expected = negative and actual = positive\n- **FN**: expected = positive and actual = negative\n- **TN**: expected = negative and actual = negative\n\nAlso compute:\n- `exact_match = 1 if actual == expected else 0`\n\n### Aggregate metrics\n\nFrom aggregate counts:\n- `accuracy = (TP + TN) / valid_rows`\n- `precision = TP / (TP + FP)`\n- `recall = TP / (TP + FN)`\n- `f1 = 2 * precision * recall / (precision + recall)`\n- `TPR = TP / (TP + FN)`\n- `TNR = TN / (TN + FP)`\n\nGuardrails:\n- if `TP + FP == 0`, precision is undefined (report null + note)\n- if `TP + FN == 0`, recall and TPR are undefined (report null + note)\n- if `TN + FP == 0`, TNR is undefined (report null + note)\n- if `precision + recall == 0`, set `f1 = 0`\n\n### Advanced quality gates\n\nBefore trusting the judge on production traffic:\n\n1. **Split integrity**: no leakage from held-out rows into prompt examples.\n2. **Confusion matrix sanity**: `TP + FP + FN + TN == valid_rows`.\n3. **Metric recomputation check**: recompute aggregate stats from row-level\n flags and compare.\n4. **TPR/TNR review**: inspect both directions for class-direction bias.\n5. **Acceptance criteria**: before the held-out test, agree on thresholds for\n the relevant metrics based on the costs of false positives and false\n negatives and whether the judge informs monitoring or automation. Do not\n infer "ship" from a universal cutoff.\n\n## 7) Report format\n\n### Simple report\n\nReturn only:\n- dataset name and dataset run URL when available\n- valid rows / total rows\n- invalid-label count\n- accuracy\n- one-sentence recommendation\n\n### Advanced report\n\nAdd:\n- confusion matrix: TP, FP, FN, TN\n- accuracy, precision, recall, F1, TPR, TNR\n- denominator notes for undefined metrics\n- top failure direction: false positives or false negatives\n- recommendation: ship, iterate, collect more labels, or do not automate\n\n## 8) Langfuse implementation notes\n\nPrefer SDK experiment evaluators for score creation. They attach item-level\nscores to the experiment traces and run-level scores to the dataset run.\n\nUse manual REST score creation only as a fallback when not using the SDK\nexperiment runner, or for local smoke tests. See\n[Scores via SDK](https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk)\nand the [Scores API reference](https://langfuse.com/docs/api) for the current\npayload shape. Before using the CLI for score creation, inspect its current\nschema and action help instead of assuming resource names, arguments, or\ncapabilities from a known package version.\n\nScore names to emit:\n\nSimple mode:\n- `judge-exact-match`\n- `judge-accuracy`\n\nAdvanced mode:\n- `judge-exact-match`\n- `judge-is-tp`\n- `judge-is-fp`\n- `judge-is-fn`\n- `judge-is-tn`\n\nRecommended metadata:\n- `expected_label`, `actual_label`\n- calibration mode: `simple` or `advanced`\n- positive/negative labels when binary metrics are used\n- evaluator prompt name+version\n- dataset/split version when used\n- run identifier\n\n## 9) Classification logic (pseudo-code)\n\nThe per-row classification each evaluator must perform:\n\n```\nnormalize(label) = strip whitespace, uppercase\nALLOWED = {POSITIVE, NEGATIVE}\n\nclassify(expected, actual):\n expected, actual = normalize(expected), normalize(actual)\n\n if expected ∉ ALLOWED or actual ∉ ALLOWED:\n mark row invalid → exclude from denominators\n\n exact_match = (expected == actual)\n is_tp = (expected == POSITIVE and actual == POSITIVE)\n is_fp = (expected == NEGATIVE and actual == POSITIVE)\n is_fn = (expected == POSITIVE and actual == NEGATIVE)\n is_tn = (expected == NEGATIVE and actual == NEGATIVE)\n```\n\n## 10) Common failure modes\n\n- label vocabulary not constrained (judge outputs free text instead of strict\n labels)\n- positive/negative label inversion between annotators and evaluator code\n- leaking `expectedOutput` into the judge task instead of only the evaluator\n- using local SDK data when the user expects a Langfuse dataset run in the UI\n- reporting only accuracy when classes are imbalanced and error direction matters\n- calculating F1 without explicit zero-denominator handling\n- using advanced validation claims without a held-out split\n\n## 11) What to do after calibration\n\n- If simple accuracy is enough: report it and stop.\n- If metrics are weak and advanced validation is needed: iterate prompt and\n few-shots on dev data only.\n- If metrics pass: freeze the baseline and monitor drift over time.\n- For qualitative diagnosis of disagreements, switch to\n `references/error-analysis.md`.',
+ "# Judge Calibration\n\n## 1. Prepare the calibration\n\nThe calibration data must:\n\n- match the information the judge receives in a real execution, based on a representative production trace, filter rules, and variable mappings;\n- include a good distribution across every possible outcome;\n- include known failure cases when badly judged traces prompted the calibration; and\n- contain expected outputs in exactly the judge's output shape and vocabulary, without passing them to the judge.\n\n1. Fetch the current docs for [LLM-as-a-Judge](https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge), [datasets](https://langfuse.com/docs/evaluation/experiments/datasets), [experiments via SDK](https://langfuse.com/docs/evaluation/experiments/experiments-via-sdk), [prompt versions and labels](https://langfuse.com/docs/prompt-management/data-model), and [LLM connections](https://langfuse.com/docs/administration/llm-connection). Use the current API schema and SDK rather than assuming fields or methods.\n2. For a new judge, immediately create its evaluator, production filters, mappings, score definition, and model, but leave its rule inactive. If no LLM connection exists, ask the user to create one and explain that the judge needs it; otherwise use the default connection unless the user specified another one.\n3. Use one dataset per judge. If it does not exist, create `judge-calibration/<judge-name>` from labeled examples or known good and bad traces. If an existing calibration dataset is not in a folder intended for future judge-calibration datasets, ask before moving it and leave it where it is if the user declines.\n4. Save the candidate prompt as `judge-calibration/<judge-name>`. If the judge was calibrated before and its prompt already exists, create a new version for the change.\n\n## 2. Run the calibration\n\nRun every calibration directly with the Langfuse Experiments SDK; do not ask the user to trigger a Prompt Experiment in the UI. Each run must:\n\n1. fetch `judge-calibration/<judge-name>` from Langfuse;\n2. fetch the candidate prompt version from `judge-calibration/<judge-name>` in Prompt Management;\n3. execute that prompt against every dataset item using the same model as the judge; and\n4. define a Boolean evaluator named `<judge-name>-judge-output-correct` and run it together with the SDK experiment. It returns `true` only when the experiment output exactly equals the dataset item's expected output.\n\nWait for every task and evaluator execution to finish before reviewing results, checking periodically while work remains pending or delayed.\n\n## 3. Review the results\n\n1. Concisely explain what you did and link to the tested prompt version and that experiment run's results screen. On the first round only, also link to the dataset and, for a new judge, the inactive judge.\n2. Inspect the accuracy and every mismatch. Treat 85% overall accuracy as quite good—about the same as human judgment—only if accuracy is also acceptable in every category; always state that benchmark and explain the mistakes, even when accuracy exceeds it.\n3. Show all mistakes in a table with columns `Dataset input`, `Expected output`, and `Actual output`.\n4. If some labels look like debatable edge cases, group those first as items whose expected labels the user may want to revisit. Otherwise, or when the user is confident in the labels, group mistakes by actual output category.\n\n## 4. Iterate until the judge is approved\n\n1. Wait for the user's response. It may be specific label guidance, a request to fix the failures, or approval of the judge.\n2. If you want to change the judge prompt, follow `references/prompt-engineering.md`. This is important. Show the prompt change to the user before applying it.\n3. After the user approves the proposed change, create a new version of `judge-calibration/<judge-name>` and rerun the SDK experiment from section 2.\n4. Wait for the run to finish, review the results again, and repeat until the user approves the judge.\n\n## 5. Deploy the approved judge\n\nOnly deploy after the user approves the judge.\n\nCopy the approved prompt into the actual evaluator, assign its Prompt Management version the `production` label, and confirm that the model, mappings, output definition, and filters match what was calibrated. For a new judge, activate its rule.",
},
{
name: "langfuse-prompt-engineering",