Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

chore(assistant): refresh embedded Langfuse skills (#17516)

langfuse-botaa05f08644fe
diff --git a/packages/langfuse-skills/src/generated/skills.js b/packages/langfuse-skills/src/generated/skills.js
index 122ec1f9b1..fecfcb23b3 100644
--- a/packages/langfuse-skills/src/generated/skills.js
+++ b/packages/langfuse-skills/src/generated/skills.js
@@ -70,7 +70,7 @@ exports.LANGFUSE_SKILLS = [
description:
"Set up evaluation by finding gaps across signal capture, monitoring, and evaluator metrics, then addressing the right one first.",
instructions:
- "# Setting up evals\n\n- **Act as executor and teacher.** Do the work and, above all, explain why the approach fits the user's goal and data.\n- **Teach decision-making, not configuration.** Explain the reasoning, tradeoffs, and limitations—not sampling rates, targets, score types, or other setup details. The response must leave the user able to repeat or change the approach and make future decisions without you.\n\nAlways read [Choosing what to evaluate](https://langfuse.com/academy/evaluate/choosing-what-to-evaluate) and [Evaluation](https://langfuse.com/academy/evaluate). Use their guidance to interpret their project, but do not return it as a generic setup plan.\n\nIf the user has a precise metric, skip to **Build the evaluator**. Otherwise, do not assume that choosing evaluator metrics is the next step.\n\n## Find the measurement gap\n\nBefore advising:\n\n- Inspect enough representative traces to present concrete findings of your own before asking about metrics.\n- Inventory existing datasets, evaluators, traces, etc. as context; do not assume they should be reused or represent a current priority. Treat their importance as unconfirmed until the user agrees.\n- Identify what is already surfaced through trace errors, logged data, existing scores, or other monitoring; what is available but not monitored; what important user or product signals are not captured; and what genuinely requires an evaluator.\n- Do not ask the user for information you can inspect.\n- If project access is unavailable, say so instead of substituting a generic plan.\n\nDo not add an evaluator that duplicates an existing error or logged signal. For example, do not evaluate valid JSON when invalid JSON already produces a trace error.\n\nTreat trace errors and logged failures as existing coverage. Unless the user wants operational monitoring, do not let them displace the search for subtler qualitative signals that are not yet captured. Do not propose alerts, paging, or release gates before asking how the measurement should be used.\n\nDo not propose evaluating a known failure with an owner and a planned one-time fix unless the user wants to track whether it recurs.\n\nIf formal error analysis is the right next step, explain why and ask whether the user wants to do it. If they agree, run it following `references/error-analysis.md`, and do not start metric selection until the analysis is complete.\n\n## Define the metric set\n\nExisting project materials inform the questions you'll ask; it does not determine what the user values.\n\nWhen forming tentative recommendations:\n\n- Start with direct evidence of user or product outcomes, recurring failures visible in traces or user reactions, and important signals the application is not yet capturing.\n- Prefer quick wins: signals that are application-specific, actionable, reasonably reliable to start measuring. A good example signal is user (dis)satisfaction.\n- Recommend metrics only for problems evidenced in current data. A behavior being common or theoretically risky is not a reason to run an evaluator.\n- For live evaluation, recommend only metrics that can be scored from the available live data without ground truth; never suggest a metric that requires knowing the correct answer, expected outcome, or ideal resolution.\n- Do not propose generic starting metrics such as `helpfulness`, `quality`, `relevance`, `hallucination`, `groundedness`, `task completion`, `task success`, or `reliability`.\n- Before presenting any metric, verify that it can actually be measured using the available data and Langfuse's supported evaluator inputs. If not, identify what must be captured instead.\n- Explain why each recommendation deserves attention before the alternatives, then ask the user whether that priority matches their goals.\n\nWhen specific failures or hard requirements are known and metric selection is the next task, ask focused, dependent questions about which ones matter and what decisions their measurement should support. Give a recommendation grounded in the project, leave the decision to the user, and wait for their answer.\n\nDo not treat any metric as selected, choose evaluator types, ask implementation questions, or implement anything until the user has answered and explicitly confirmed the metric set.\n\n## Build the evaluator\n\nFor evaluator functionality, use the unstable API endpoints.\n\nFollow [Writing good evaluators](https://langfuse.com/academy/evaluate/writing-evaluators) to choose the evaluator type; do not always default to an LLM-as-a-judge.\n\n- Before creating the evaluator, verify its target filter by fetching the observations it actually matches. Confirm the set is exactly what you intend—no duplicates that would be scored (and billed) twice.\n- Prioritize a clean score name: it is the metric that lands on every observation, so name it after what is measured (`refusal`), not after the evaluator (`refusal judge`). Keep any evaluator-mechanism wording out of the score name.\n- Explain why the chosen evaluation method measures the intended behavior, what evidence it relies on, what it cannot tell the user, and when another method would be better.\n- When methods have significant trade-offs and none is clearly superior, explain the options and let the user decide before implementation.\n- Only if an LLM-as-a-judge is the best fit, calibrate it on real examples before treating it as ready(`references/judge-calibration.md`). You can do this by running an experiment on a dataset where the prompt being tested is the LLM-as-a-judge prompt.\n- Share a link to the evaluator.",
+ "# Setting up evals\n\nAct as a PM, analyst, teacher, and at the end, an executor. The bulk of the work consists of probing the user to really understand what they are looking for, and get a clear picture of what would serve them best. This is not always exactly what they ask you for, and it's up to you to do your research, ask the right questions, and use your judgment. Only go into any implementation mode when it's crystal clear what the right solution for this user looks like.\n\n## 1. Determine Online or Offline\n\nDetermine whether the user is looking for online or offline evaluation. If the goal is unclear, ask which decision the user is trying to make. If they need both, establish which one to address first.\n\n## 2. Establish the Metric Set\n\nIf the user supplied metrics, review them against the application, available data, and intended decision. A useful metric names a specific observable behavior, is measurable, and changes a decision. Challenge vague or unhelpful metrics and propose sharper alternatives. Read [Choosing what to evaluate](https://langfuse.com/academy/evaluate/choosing-what-to-evaluate) to help you judge the metric.\n\nIf the user did not supply metrics:\n\n- **Online:** The user likely doesn't know what they want. Execute the complete [User Feedback](user-feedback.md) workflow, backfill derivable signals where possible. Explain to the user why this is the right step to take. People often say they want evals without knowing what they actually need. This is one such case. You are completing their request by executing the user-feedback workflow.\n- **Offline:** inspect existing datasets and experiment results, then ask questions to determine:\n 1. is the user happy with their datasets, or do they want to evaluate something they don't have a dataset for?\n 2. if they want to use an existing dataset, do they already manually review experiment outputs on that dataset, and if yes, what do they look for?\n These questions should help narrow down what's important to the user and what you should focus on. Read [Choosing what to evaluate](https://langfuse.com/academy/evaluate/choosing-what-to-evaluate) to come up with high quality metrics to evaluate.\n\n\nPresent the resulting metric set as one table with: Priority; Status (existing or new); Metric; Source or evidence; Why it matters and what decision it informs; and Measurability. Revise it with the user and do not proceed until they explicitly confirm it.\n\n## 3. Implement and Verify\n\nAfter the metric set is confirmed:\n\n- **Online:** follow [Writing good evaluators](https://langfuse.com/academy/evaluate/writing-evaluators), then implement and verify the online evaluators.\n- **Offline:** confirm that an appropriate dataset exists. If not, execute [Dataset Construction](create-dataset.md). Then implement the evaluators and run the experiment.\n\nDo not default to an LLM-as-a-judge. When methods have significant trade-offs and none is clearly superior, present the options and let the user decide.\n\nFor evaluator functionality, use the unstable API endpoints.\n\n- Before creating an evaluator, fetch the observations matched by its target filter and confirm that they are correct and will not be scored twice.\n- Name the score after what is measured (`refusal`), not the evaluator (`refusal judge`).\n- If an LLM-as-a-judge is the best fit, calibrate it on real examples before treating it as ready.\n- Share a link to each evaluator.",
},
{
name: "langfuse-skill-feedback",
@@ -89,9 +89,9 @@ exports.LANGFUSE_SKILLS = [
{
name: "langfuse-user-feedback",
description:
- "Wires up user feedback (thumbs up/down, ratings, comments) from an application's frontend to Langfuse scores. Use when user wants to capture end-user feedback, add ratings to traces, or connect user complaints to Langfuse.",
+ "Capture user feedback as Langfuse scores. Use when choosing or implementing explicit ratings, behavioral signals, conversation signals, or task outcomes.",
instructions:
- "# User Feedback\n\nTracing must already be set up — feedback is stored as scores on traces.\n\nDocs: https://langfuse.com/docs/observability/features/user-feedback\n\n## Workflow\n\n### 1. Determine What Feedback to Capture\n\nIf the user has asked for something specific, go with that. Otherwise, look at the application and **present a few UX options** for how feedback could work, then ask the user which they prefer before implementing.\n\nCommon UX patterns to suggest:\n\n| UX Pattern | Best for | How it works |\n|------------|----------|--------------|\n| Thumbs up/down | Chat apps, Q&A | Simple binary buttons next to each response |\n| Star rating (1–5) | Content generation, summaries | Star row or dropdown after each output |\n| \"Was this helpful?\" banner | Search, documentation assistants | Single yes/no prompt at the bottom of a response |\n| Regenerate / copy tracking | Any app with these actions | Implicit — log when users retry (negative signal) or copy output (positive signal) |\n| Free-text comment | Complex outputs, internal tools | Optional text field alongside a rating |\n| Report button | Any user-facing app | Flag icon to report bad/harmful responses |\n\nThis table is not exhaustive — if the application suggests a different feedback pattern that fits better, propose that instead. Present 2–3 options that match the application's use case and ask the user which approach they'd like. This decision shapes everything downstream (score names, data types, frontend components), so it's important to align early.\n\nFeedback can be **explicit** (user rates via thumbs, stars, etc.) or **implicit** (derived from behavior like copying output, retrying, or escalating to support). Both are stored as scores. Explicit feedback requires the trace ID to reach the frontend; implicit feedback is logged server-side where the event already happens.\n\n### 2. Choose Score Names\n\nName reflects the signal source, not what you hope it measures (e.g., `user-thumbs` not `response-quality` — a thumbs down doesn't tell you *what* was wrong). Avoid generic names like `feedback` or `score`.\n\nRules:\n- Lowercase with hyphens\n- One consistent name per feedback type across the entire app\n- If capturing multiple signals, each gets its own distinct name\n\n### 3. Implement Score Creation\n\nFetch and follow the current [user feedback guide](https://langfuse.com/docs/observability/features/user-feedback) and [score ingestion docs](https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk) before editing code. They contain the current browser and server SDK APIs plus framework-specific patterns for returning trace IDs to the frontend.\n\n**For implicit feedback (server-side):** Create the score where the behavior is already handled in application code.\n\n**For explicit feedback (frontend):** Make the relevant trace ID available to the frontend and use the current Langfuse browser SDK with a public key only. Never expose a Langfuse secret key in browser code.\n\n### 4. Verify\n\nTrigger a feedback action and check the trace's Scores tab in Langfuse. Confirm the score name, value, and data type are correct.\n\nPoint users to what they can do with feedback data: filter traces by low scores, use score analytics for trends, build annotation queues for team review.\n\n## Common Mistakes\n\n| Mistake | Problem | Fix |\n|---------|---------|-----|\n| Secret key in frontend code | Security risk | Use the current browser SDK with a public key only |\n| Missing `dataType` on boolean scores | Value `1` inferred as `NUMERIC` | Always pass `dataType: \"BOOLEAN\"` explicitly |\n| Inconsistent score names across the app | Can't aggregate or filter reliably | Pick one name per feedback type, use it everywhere |",
+ "# User Feedback\n\n## Workflow\n\n### 1. Determine What Feedback to Capture\n\nIf the user has asked for something specific, go with that. Otherwise, look at the application and **present a few UX options** for how feedback could work, then ask the user which they prefer before implementing.\n\nBefore presenting options, read [Capturing signals](https://langfuse.com/academy/monitoring/capturing-signals) and inspect the relevant application flow and existing scores. Present findings specific to the application rather than repeating examples from the docs.\n\nPresent every set of proposed signals or metrics as a table with:\n\n- Priority: P0, P1, ..\n- Status: whether the score or signal is already implemented or newly proposed\n- Signal name: Name the score after the observed signal, not the quality you hope it represents: draft_edited, not quality. Reuse the same name for the same signal throughout the application.\n- Explanation: what it indicates, its limitations and biases, and why it fits the application\n- Effort: XS-L\n- Implementation: a very short description such as “Langfuse evaluator” or “application code”. When a signal can be captured deterministically in both application code and a Langfuse code evaluator, present both as options.\n\nWhen multiple signals can be captured by one evaluator, group them and reflect this in the Implementation and Effort columns.\n\nWhen proposing an outcome signal, explicitly include the implementation complexity in its effort and explanation; unless the supporting workflow and data already exist or the expected value clearly justifies building them, recommend deferring it.\n\n### 2. Implement the Feedback\n\nRead the [user feedback loop guide](https://langfuse.com/guides/user-feedback-loop) and follow its links to the current SDK documentation before editing code.\n\n### 3. Verify\n\nTrigger each implemented signal and confirm that its score has the intended name, value, data type, and attachment.",
},
{
name: "langfuse-v4-project-migration",