Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

chore(assistant): refresh embedded Langfuse skills (#17072)

langfuse-bot26f15d39086f
diff --git a/packages/langfuse-skills/src/generated/skills.js b/packages/langfuse-skills/src/generated/skills.js
index 4317bc9fd6..122ec1f9b1 100644
--- a/packages/langfuse-skills/src/generated/skills.js
+++ b/packages/langfuse-skills/src/generated/skills.js
@@ -35,7 +35,7 @@ exports.LANGFUSE_SKILLS = [
description:
"Instrument LLM applications with Langfuse tracing. Use when setting up Langfuse, adding observability to LLM calls, or auditing existing instrumentation.",
instructions:
- "# Langfuse Observability\n\nInstrument LLM applications with Langfuse tracing, following best practices and tailored to your use case.\n\n## Workflow\n\n### 1. Assess Current State\n\nCheck the project:\n\n- Is Langfuse SDK installed?\n- What LLM frameworks are used? (OpenAI SDK, LangChain, LlamaIndex, Vercel AI SDK, etc.)\n- Is there existing instrumentation?\n\n**No integration yet:** Set up Langfuse using a framework integration if available. Integrations capture more context automatically and require less code than manual instrumentation.\n\n**Integration exists:** Audit against baseline requirements below.\n\n### 2. Verify Baseline Requirements\n\nEvery trace should have these fundamentals:\n\n| Requirement | Check | Why |\n| ------------------------- | ---------------------------------------------------------------------------------------- | ------------------------------------------------------ |\n| Model name | Is the LLM model captured? | Enables model comparison and filtering |\n| Token usage | Are input/output tokens tracked? | Enables automatic cost calculation |\n| Good trace names | Are names descriptive? (`chat-response`, not `trace-1`) | Makes traces findable and filterable |\n| Span hierarchy | Are multi-step operations nested properly? | Shows which step is slow or failing |\n| Correct observation types | Are generations marked as generations, and is each other call given its most specific type (`retriever` for a lookup, `agent` for a subagent, etc.) rather than a generic `tool`/`span`? See the [observation types docs](https://langfuse.com/docs/observability/features/observation-types). | Enables model-specific analytics and drives the Agent Graph |\n| Sensitive data masked | Is PII/confidential data excluded or masked? | Prevents data leakage |\n| Trace input/output | Does the trace capture meaningful input/output? Is input explicitly set to show only relevant data (e.g., user message), not all function args? | Makes traces readable in the UI and avoids leaking sensitive args |\n\nFramework integrations (OpenAI, LangChain, etc.) handle model name, tokens, and observation types automatically. Prefer integrations over manual instrumentation.\n\nDocs: https://langfuse.com/docs/tracing\n\n**Beyond the baseline**, add context relevant to the app. Infer from code where possible; only ask when it's not obvious (e.g. how they judge a good vs. bad response, what they'd filter a dashboard by, which user segments they'd compare). These are not baseline — add only what fits.\n\n| If code shows... | Add | Why |\n| ---------------------------------------------------- | ------------------- | ------------------------------------------------------------------------------- |\n| Conversation history, chat endpoints, message arrays | `session_id` | Groups conversations — [docs](https://langfuse.com/docs/tracing-features/sessions) |\n| User authentication, `user_id` variables | `user_id` | User filtering and cost attribution — [docs](https://langfuse.com/docs/tracing-features/users) |\n| Multiple distinct endpoints/features | `feature` tag | Per-feature analytics — [docs](https://langfuse.com/docs/tracing-features/tags) |\n| Customer/tenant identifiers | `customer_tier` tag | Cost/quality breakdown by segment — [docs](https://langfuse.com/docs/tracing-features/tags) |\n| Feedback collection, ratings | Feedback score | Quality filtering and trends — [docs](https://langfuse.com/docs/scores/overview) |\n\n### 3. Run and Self-Audit the Traces (required)\n\nInstrumentation isn't done when the code compiles. This is a loop you own as the agent, it's your responsibility to deliver traces of the highest quality your can produce:\n\n**a.** Execute the instrumented path end-to-end so a trace is actually sent.\n\n**b.** Fetch the trace(s) you just created from Langfuse. Any method works (`langfuse-cli`, REST API, SDK, MCP); the CLI is usually simplest — see [references/cli.md](references/cli.md).\n\n**c.** Audit the trace against the best-practices page. **Always fetch it fresh — never audit from memory, this cannot be skipped** (the guidance changes over time):\n\nhttps://langfuse.com/docs/observability/best-practices\n\nAsk yourself, for each observation: is all data that a user might need in the future, to understand exactly what context the agent had when it made decisions, available in Langfuse?\n\n**d.** Fix every gap you find, then re-run and re-fetch to confirm. Repeat until the trace clears the guidance. Then report what you audited and changed, and link the final trace.\n\n### 4. Explore Traces With the User\n\nWith a clean trace in place, invite the user to explore it in the Langfuse UI:\n\n\"Your traces are now appearing in Langfuse. Take a look at a few of them—see what data is being captured, what's useful, and what's missing.\"\n\nPoint them to the relevant views:\n\n- Traces view: individual requests\n- Sessions view: grouped conversations (if `session_id` added)\n- Dashboard: filtered views using tags\n- Scores: filter by quality metrics\n\nThis helps them understand what they're getting, spot what's missing, and ask better questions about what to add next.\n\n## Always Explain Why\n\nWhen suggesting additions, explain the user benefit:\n\n```\n\"I recommend adding session_id to your traces.\n\nWhy: This groups messages from the same conversation together.\nYou'll be able to see full conversation flows in the Sessions view,\nmaking it much easier to debug multi-turn interactions.\n\nLearn more: https://langfuse.com/docs/tracing-features/sessions\"\n```\n\n## Multi-agent systems (subagent dispatch)\n\nWhen one agent's execution dispatches OTHER agents (coding agents like Claude Code/Codex, research agents, orchestrator/worker architectures), a few extra points on top of the baseline:\n\n- **Type a subagent's own execution as `agent`, not `tool`/`span`.** A bare tool/span for a dispatch hides all of the subagent's internal structure; `agent` lets it show up as its own node in the [Agent Graph](https://langfuse.com/docs/observability/features/agent-graphs).\n- **Don't emit duplicate dispatch + execution nodes.** Emitting both a `tool`-typed \"dispatch\" span and a separate `agent`-typed observation for the same subagent, as siblings, double-represents one event. Emit only the `agent` observation when you have the subagent's actual execution; keep a bare tool span only as a fallback when you have no visibility into what the subagent did.\n- **Nest recursively.** Nest the subagent's `agent` observation under the `agent` or `span` that orchestrates the dispatch, as a sibling of the `generation` that requested it. Within the subagent, put its generations, tool calls, and nested subagent executions under that subagent's `agent` observation; each tool or nested subagent is a sibling of the generation that requested it, not a child of that generation.\n- **Name subagents distinctly.** Frameworks often default every subagent to the same generic role name, making them indistinguishable in the tree and graph (nodes key on name). Derive a specific name from the subagent's actual task/role when the framework doesn't provide one.\n\n## Common Mistakes\n\n| Mistake | Problem | Fix |\n| ---------------------------------------------- | --------------------------------------------------- | --------------------------------------------------------------------------------- |\n| No `flush()` in scripts | Traces never sent | Call `langfuse.flush()` before exit |\n| Flat traces | Can't see which step failed | Use nested spans for distinct steps |\n| Generic trace names | Hard to filter | Use descriptive names: `chat-response`, `doc-summary` |\n| Logging sensitive data | Data leakage risk | Mask PII before tracing |\n| Not explicitly setting input with `@observe` | All function args become trace input (including API keys, configs) | Python: use `langfuse.update_current_span(input=...)`. JS/TS: use `updateActiveObservation({ input: ... })`. Set only the relevant input (e.g., user message) |\n| Manual instrumentation when integration exists | More code, less context | Use framework integration |\n| Langfuse import before env vars loaded | Langfuse initializes with missing/wrong credentials | Import Langfuse AFTER loading environment variables (e.g., after `load_dotenv()`) |\n| Wrong import order with OpenAI | Langfuse can't patch the OpenAI client | Import Langfuse and call its setup BEFORE importing OpenAI client |",
+ "# Langfuse Observability\n\nInstrument LLM applications with Langfuse tracing, following best practices and tailored to your use case.\n\n## Workflow\n\n### 1. Assess Current State\n\nCheck the project:\n\n- Is Langfuse SDK installed?\n- What LLM frameworks are used? (OpenAI SDK, LangChain, LlamaIndex, Vercel AI SDK, etc.)\n- Is there existing instrumentation?\n\n**No integration yet:** Set up Langfuse using a framework integration if available. Integrations capture more context automatically and require less code than manual instrumentation.\n\n**Integration exists:** Audit against baseline requirements below.\n\n### 2. Verify Baseline Requirements\n\nEvery trace should have these fundamentals:\n\n| Requirement | Check | Why |\n| ------------------------- | ---------------------------------------------------------------------------------------- | ------------------------------------------------------ |\n| Model name | Is the LLM model captured? | Enables model comparison and filtering |\n| Token usage | Are input/output tokens tracked? | Enables automatic cost calculation |\n| Good trace names | Are names descriptive? (`chat-response`, not `trace-1`) | Makes traces findable and filterable |\n| Span hierarchy | Are multi-step operations nested properly? | Shows which step is slow or failing |\n| Correct observation types | Are generations marked as generations, and is each other call given its most specific type (`retriever` for a lookup, `agent` for a subagent, etc.) rather than a generic `tool`/`span`? See the [observation types docs](https://langfuse.com/docs/observability/features/observation-types). | Enables model-specific analytics and drives the Agent Graph |\n| Sensitive data masked | Is PII/confidential data excluded or masked? | Prevents data leakage |\n| Trace input/output | Does the trace capture meaningful input/output? Is input explicitly set to show only relevant data (e.g., user message), not all function args? | Makes traces readable in the UI and avoids leaking sensitive args |\n\nFramework integrations (OpenAI, LangChain, etc.) handle model name, tokens, and observation types automatically. Prefer integrations over manual instrumentation.\n\nDocs: https://langfuse.com/docs/tracing\n\n**Beyond the baseline**, add context relevant to the app. Infer from code where possible; only ask when it's not obvious (e.g. how they judge a good vs. bad response, what they'd filter a dashboard by, which user segments they'd compare). These are not baseline — add only what fits.\n\n| If code shows... | Add | Why |\n| ---------------------------------------------------- | ------------------- | ------------------------------------------------------------------------------- |\n| Conversation history, chat endpoints, message arrays | `session_id` | Groups conversations — [docs](https://langfuse.com/docs/tracing-features/sessions) |\n| User authentication, `user_id` variables | `user_id` | User filtering and cost attribution — [docs](https://langfuse.com/docs/tracing-features/users) |\n| Multiple distinct endpoints/features | `feature` tag | Per-feature analytics — [docs](https://langfuse.com/docs/tracing-features/tags) |\n| Customer/tenant identifiers | `customer_tier` tag | Cost/quality breakdown by segment — [docs](https://langfuse.com/docs/tracing-features/tags) |\n| Feedback collection, ratings | Feedback score | Quality filtering and trends — [docs](https://langfuse.com/docs/scores/overview) |\n| Image/audio/file input or output (e.g. image generation, vision) | Media in observation input/output | Renders media in the UI; base64 data URIs and external URLs are handled automatically, wrap raw bytes/files in `LangfuseMedia`. Follow the [multi-modality docs](https://langfuse.com/docs/observability/features/multi-modality) to implement |\n\n### 3. Run and Self-Audit the Traces (required)\n\nInstrumentation isn't done when the code compiles. This is a loop you own as the agent, it's your responsibility to deliver traces of the highest quality your can produce:\n\n**a.** Execute the instrumented path end-to-end so a trace is actually sent.\n\n**b.** Fetch the trace(s) you just created from Langfuse. Any method works (`langfuse-cli`, REST API, SDK, MCP); the CLI is usually simplest — see [references/cli.md](references/cli.md).\n\n**c.** Audit the trace against the best-practices page. **Always fetch it fresh — never audit from memory, this cannot be skipped** (the guidance changes over time):\n\nhttps://langfuse.com/docs/observability/best-practices\n\nAsk yourself, for each observation: is all data that a user might need in the future, to understand exactly what context the agent had when it made decisions, available in Langfuse?\n\n**d.** Fix every gap you find, then re-run and re-fetch to confirm. Repeat until the trace clears the guidance. Then report what you audited and changed, and link the final trace.\n\n### 4. Explore Traces With the User\n\nWith a clean trace in place, invite the user to explore it in the Langfuse UI:\n\n\"Your traces are now appearing in Langfuse. Take a look at a few of them—see what data is being captured, what's useful, and what's missing.\"\n\nPoint them to the relevant views:\n\n- Traces view: individual requests\n- Sessions view: grouped conversations (if `session_id` added)\n- Dashboard: filtered views using tags\n- Scores: filter by quality metrics\n\nThis helps them understand what they're getting, spot what's missing, and ask better questions about what to add next.\n\n## Always Explain Why\n\nWhen suggesting additions, explain the user benefit:\n\n```\n\"I recommend adding session_id to your traces.\n\nWhy: This groups messages from the same conversation together.\nYou'll be able to see full conversation flows in the Sessions view,\nmaking it much easier to debug multi-turn interactions.\n\nLearn more: https://langfuse.com/docs/tracing-features/sessions\"\n```\n\n## Multi-agent systems (subagent dispatch)\n\nWhen one agent's execution dispatches OTHER agents (coding agents like Claude Code/Codex, research agents, orchestrator/worker architectures), a few extra points on top of the baseline:\n\n- **Type a subagent's own execution as `agent`, not `tool`/`span`.** A bare tool/span for a dispatch hides all of the subagent's internal structure; `agent` lets it show up as its own node in the [Agent Graph](https://langfuse.com/docs/observability/features/agent-graphs).\n- **Don't emit duplicate dispatch + execution nodes.** Emitting both a `tool`-typed \"dispatch\" span and a separate `agent`-typed observation for the same subagent, as siblings, double-represents one event. Emit only the `agent` observation when you have the subagent's actual execution; keep a bare tool span only as a fallback when you have no visibility into what the subagent did.\n- **Nest recursively.** Nest the subagent's `agent` observation under the `agent` or `span` that orchestrates the dispatch, as a sibling of the `generation` that requested it. Within the subagent, put its generations, tool calls, and nested subagent executions under that subagent's `agent` observation; each tool or nested subagent is a sibling of the generation that requested it, not a child of that generation.\n- **Name subagents distinctly.** Frameworks often default every subagent to the same generic role name, making them indistinguishable in the tree and graph (nodes key on name). Derive a specific name from the subagent's actual task/role when the framework doesn't provide one.\n\n## Common Mistakes\n\n| Mistake | Problem | Fix |\n| ---------------------------------------------- | --------------------------------------------------- | --------------------------------------------------------------------------------- |\n| No `flush()` in scripts | Traces never sent | Call `langfuse.flush()` before exit |\n| Flat traces | Can't see which step failed | Use nested spans for distinct steps |\n| Generic trace names | Hard to filter | Use descriptive names: `chat-response`, `doc-summary` |\n| Logging sensitive data | Data leakage risk | Mask PII before tracing |\n| Not explicitly setting input with `@observe` | All function args become trace input (including API keys, configs) | Python: use `langfuse.update_current_span(input=...)`. JS/TS: use `updateActiveObservation({ input: ... })`. Set only the relevant input (e.g., user message) |\n| Manual instrumentation when integration exists | More code, less context | Use framework integration |\n| Langfuse import before env vars loaded | Langfuse initializes with missing/wrong credentials | Import Langfuse AFTER loading environment variables (e.g., after `load_dotenv()`) |\n| Wrong import order with OpenAI | Langfuse can't patch the OpenAI client | Import Langfuse and call its setup BEFORE importing OpenAI client |",
},
{
name: "langfuse-judge-calibration",
@@ -70,7 +70,7 @@ exports.LANGFUSE_SKILLS = [
description:
"Set up evaluation by finding gaps across signal capture, monitoring, and evaluator metrics, then addressing the right one first.",
instructions:
- "# Setting up evals\n\n- **Act as executor and teacher.** Do the work and, above all, explain why the approach fits the user's goal and data.\n- **Teach decision-making, not configuration.** Explain the reasoning, tradeoffs, and limitations—not sampling rates, targets, score types, or other setup details. The response must leave the user able to repeat or change the approach and make future decisions without you.\n\nAlways read [Choosing what to evaluate](https://langfuse.com/academy/evaluate/choosing-what-to-evaluate) and [Evaluation](https://langfuse.com/academy/evaluate). Use their guidance to interpret their project, but do not return it as a generic setup plan.\n\nIf the user has a precise metric, skip to **Build the evaluator**. Otherwise, do not assume that choosing evaluator metrics is the next step.\n\n## Find the measurement gap\n\nBefore advising:\n\n- Inspect enough representative traces to present concrete findings of your own before asking about metrics.\n- Inventory existing datasets, evaluators, traces, etc. as context; do not assume they should be reused or represent a current priority. Treat their importance as unconfirmed until the user agrees.\n- Identify what is already surfaced through trace errors, logged data, existing scores, or other monitoring; what is available but not monitored; what important user or product signals are not captured; and what genuinely requires an evaluator.\n- Do not ask the user for information you can inspect.\n- If project access is unavailable, say so instead of substituting a generic plan.\n\nDo not add an evaluator that duplicates an existing error or logged signal. For example, do not evaluate valid JSON when invalid JSON already produces a trace error.\n\nTreat trace errors and logged failures as existing coverage. Unless the user wants operational monitoring, do not let them displace the search for subtler qualitative signals that are not yet captured. Do not propose alerts, paging, or release gates before asking how the measurement should be used.\n\nDo not propose evaluating a known failure with an owner and a planned one-time fix unless the user wants to track whether it recurs.\n\nIf formal error analysis is the right next step, explain why and ask whether the user wants to do it. If they agree, run it following `references/error-analysis.md`, and do not start metric selection until the analysis is complete.\n\n## Define the metric set\n\nExisting project materials inform the questions you'll ask; it does not determine what the user values.\n\nWhen forming tentative recommendations:\n\n- Start with direct evidence of user or product outcomes, recurring failures visible in traces or user reactions, and important signals the application is not yet capturing.\n- Prefer quick wins: signals that are application-specific, actionable, reasonably reliable to start measuring. A good example signal is user (dis)satisfaction.\n- Recommend metrics only for problems evidenced in current data. A behavior being common or theoretically risky is not a reason to run an evaluator.\n- For live evaluation, recommend only metrics that can be scored from the available live data without ground truth; never suggest a metric that requires knowing the correct answer, expected outcome, or ideal resolution.\n- Do not propose generic starting metrics such as `helpfulness`, `quality`, `relevance`, `hallucination`, `groundedness`, `task completion`, `task success`, or `reliability`.\n- Before presenting any metric, verify that it can actually be measured using the available data and Langfuse's supported evaluator inputs. If not, identify what must be captured instead.\n- Explain why each recommendation deserves attention before the alternatives, then ask the user whether that priority matches their goals.\n\nWhen specific failures or hard requirements are known and metric selection is the next task, ask focused, dependent questions about which ones matter and what decisions their measurement should support. Give a recommendation grounded in the project, leave the decision to the user, and wait for their answer.\n\nDo not treat any metric as selected, choose evaluator types, ask implementation questions, or implement anything until the user has answered and explicitly confirmed the metric set.\n\n## Build the evaluator\n\nFor evaluator functionality, use the unstable API endpoints.\n\nFollow [Writing good evaluators](https://langfuse.com/academy/evaluate/writing-evaluators) to choose the evaluator type; do not always default to an LLM-as-a-judge.\n\n- Explain why the chosen evaluation method measures the intended behavior, what evidence it relies on, what it cannot tell the user, and when another method would be better.\n- When methods have significant trade-offs and none is clearly superior, explain the options and let the user decide before implementation.\n- Only if an LLM-as-a-judge is the best fit, calibrate it on real examples before treating it as ready(`references/judge-calibration.md`). You can do this by running an experiment on a dataset where the prompt being tested is the LLM-as-a-judge prompt.\n- Share a link to the evaluator.",
+ "# Setting up evals\n\n- **Act as executor and teacher.** Do the work and, above all, explain why the approach fits the user's goal and data.\n- **Teach decision-making, not configuration.** Explain the reasoning, tradeoffs, and limitations—not sampling rates, targets, score types, or other setup details. The response must leave the user able to repeat or change the approach and make future decisions without you.\n\nAlways read [Choosing what to evaluate](https://langfuse.com/academy/evaluate/choosing-what-to-evaluate) and [Evaluation](https://langfuse.com/academy/evaluate). Use their guidance to interpret their project, but do not return it as a generic setup plan.\n\nIf the user has a precise metric, skip to **Build the evaluator**. Otherwise, do not assume that choosing evaluator metrics is the next step.\n\n## Find the measurement gap\n\nBefore advising:\n\n- Inspect enough representative traces to present concrete findings of your own before asking about metrics.\n- Inventory existing datasets, evaluators, traces, etc. as context; do not assume they should be reused or represent a current priority. Treat their importance as unconfirmed until the user agrees.\n- Identify what is already surfaced through trace errors, logged data, existing scores, or other monitoring; what is available but not monitored; what important user or product signals are not captured; and what genuinely requires an evaluator.\n- Do not ask the user for information you can inspect.\n- If project access is unavailable, say so instead of substituting a generic plan.\n\nDo not add an evaluator that duplicates an existing error or logged signal. For example, do not evaluate valid JSON when invalid JSON already produces a trace error.\n\nTreat trace errors and logged failures as existing coverage. Unless the user wants operational monitoring, do not let them displace the search for subtler qualitative signals that are not yet captured. Do not propose alerts, paging, or release gates before asking how the measurement should be used.\n\nDo not propose evaluating a known failure with an owner and a planned one-time fix unless the user wants to track whether it recurs.\n\nIf formal error analysis is the right next step, explain why and ask whether the user wants to do it. If they agree, run it following `references/error-analysis.md`, and do not start metric selection until the analysis is complete.\n\n## Define the metric set\n\nExisting project materials inform the questions you'll ask; it does not determine what the user values.\n\nWhen forming tentative recommendations:\n\n- Start with direct evidence of user or product outcomes, recurring failures visible in traces or user reactions, and important signals the application is not yet capturing.\n- Prefer quick wins: signals that are application-specific, actionable, reasonably reliable to start measuring. A good example signal is user (dis)satisfaction.\n- Recommend metrics only for problems evidenced in current data. A behavior being common or theoretically risky is not a reason to run an evaluator.\n- For live evaluation, recommend only metrics that can be scored from the available live data without ground truth; never suggest a metric that requires knowing the correct answer, expected outcome, or ideal resolution.\n- Do not propose generic starting metrics such as `helpfulness`, `quality`, `relevance`, `hallucination`, `groundedness`, `task completion`, `task success`, or `reliability`.\n- Before presenting any metric, verify that it can actually be measured using the available data and Langfuse's supported evaluator inputs. If not, identify what must be captured instead.\n- Explain why each recommendation deserves attention before the alternatives, then ask the user whether that priority matches their goals.\n\nWhen specific failures or hard requirements are known and metric selection is the next task, ask focused, dependent questions about which ones matter and what decisions their measurement should support. Give a recommendation grounded in the project, leave the decision to the user, and wait for their answer.\n\nDo not treat any metric as selected, choose evaluator types, ask implementation questions, or implement anything until the user has answered and explicitly confirmed the metric set.\n\n## Build the evaluator\n\nFor evaluator functionality, use the unstable API endpoints.\n\nFollow [Writing good evaluators](https://langfuse.com/academy/evaluate/writing-evaluators) to choose the evaluator type; do not always default to an LLM-as-a-judge.\n\n- Before creating the evaluator, verify its target filter by fetching the observations it actually matches. Confirm the set is exactly what you intend—no duplicates that would be scored (and billed) twice.\n- Prioritize a clean score name: it is the metric that lands on every observation, so name it after what is measured (`refusal`), not after the evaluator (`refusal judge`). Keep any evaluator-mechanism wording out of the score name.\n- Explain why the chosen evaluation method measures the intended behavior, what evidence it relies on, what it cannot tell the user, and when another method would be better.\n- When methods have significant trade-offs and none is clearly superior, explain the options and let the user decide before implementation.\n- Only if an LLM-as-a-judge is the best fit, calibrate it on real examples before treating it as ready(`references/judge-calibration.md`). You can do this by running an experiment on a dataset where the prompt being tested is the LLM-as-a-judge prompt.\n- Share a link to the evaluator.",
},
{
name: "langfuse-skill-feedback",