Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(aio): point agents reading traces to online evals (#128267)

Bernat Torresd232caf1769f
diff --git a/products/ai_observability/mcp/prompts/query-llm-trace.md b/products/ai_observability/mcp/prompts/query-llm-trace.md
index 0ccfca269399..97c4558f86af 100644
--- a/products/ai_observability/mcp/prompts/query-llm-trace.md
+++ b/products/ai_observability/mcp/prompts/query-llm-trace.md
@@ -106,3 +106,4 @@ To protect the agent's context window, very large traces are compacted before th
- The `events` array contains ALL events in the trace (including deeply nested ones), making this suitable for full tree reconstruction.
- Use `query-llm-traces-list` first to find traces, then this tool to inspect a specific one.
- Only ask for `detail: "full"` on a trace you have already narrowed down. A full-detail read of an unremarkable trace spends context you will need later.
+- When the traces show a failure that keeps happening, offer to score new traces for it automatically with an online evaluation. Load the `creating-online-evaluations` skill to set one up.
diff --git a/products/ai_observability/mcp/prompts/query-llm-traces-list.md b/products/ai_observability/mcp/prompts/query-llm-traces-list.md
index ce51049cc9a5..82710583c31c 100644
--- a/products/ai_observability/mcp/prompts/query-llm-traces-list.md
+++ b/products/ai_observability/mcp/prompts/query-llm-traces-list.md
@@ -215,3 +215,4 @@ Use `limit` and `offset` for pagination. The default limit is 100. The response
- Use `filterTestAccounts: true` by default to exclude internal users unless the user asks otherwise.
- The default time range is last 7 days. LLM trace data tends to be recent, so shorter ranges are usually appropriate.
- For deep inspection of a single trace (full event tree with all nested children and complete properties), use `query-llm-trace` with the trace's `id`.
+- When the traces show a failure that keeps happening, offer to score new traces for it automatically with an online evaluation. Load the `creating-online-evaluations` skill to set one up.
diff --git a/products/ai_observability/skills/creating-online-evaluations/SKILL.md b/products/ai_observability/skills/creating-online-evaluations/SKILL.md
index 28ca58194a14..569375072dde 100644
--- a/products/ai_observability/skills/creating-online-evaluations/SKILL.md
+++ b/products/ai_observability/skills/creating-online-evaluations/SKILL.md
@@ -4,10 +4,9 @@ description: >
Author continuously-running online evaluations in PostHog AI observability, grounded in real failure
modes you've identified. Use when the user wants evaluations that automatically score new generations or
whole traces going forward — "create an eval to catch X", "continuously check that responses do Y",
- "turn these failures into evals". Covers letting the explored data decide how many evals to create,
+ "turn these failures into evals", or after reading traces finds a recurring failure. Covers letting the explored data decide how many evals to create,
proposing that set for the user to pick, choosing the target and eval type (hog / llm_judge /
- sentiment), configuring a provider and model for an llm_judge eval (a provider key gates enabling, not
- creation), scoping which generations trigger it via conditions, creating disabled, verifying scope,
+ sentiment), configuring an llm_judge's provider and model, scoping which generations trigger it via conditions, creating disabled, verifying scope,
enabling, and backfilling past data. Proposes a sentiment eval when no failure mode is worth catching. Finding and ranking the
failure modes worth evaluating is its own job — use exploring-ai-failures first. To debug or manage
evaluations that already exist, use exploring-llm-evaluations.
diff --git a/products/ai_observability/skills/exploring-ai-failures/SKILL.md b/products/ai_observability/skills/exploring-ai-failures/SKILL.md
index 78c02bc34f68..a4db7ecb435d 100644
--- a/products/ai_observability/skills/exploring-ai-failures/SKILL.md
+++ b/products/ai_observability/skills/exploring-ai-failures/SKILL.md
@@ -118,7 +118,8 @@ example trace deep links** on your own — don't wait to be asked, and don't mak
You read these traces, but you can misread one — a trace that looks like a hallucination may be correct in
context, and some of what you flag will be you misunderstanding the trace, not a real failure. So don't
present the list as settled fact. Give the user a couple of linked examples per mode, ask them to open the
-links, then ask **which mode they want to focus on** next.
+links, then ask **which mode they want to focus on** next. Once they pick one, offer to catch it on new traffic
+with an online evaluation (`creating-online-evaluations`).
(A list assembled from error messages or metric counts you never read is the loud subset, not this — go
back to Step 3.)
diff --git a/products/ai_observability/skills/exploring-llm-clusters/SKILL.md b/products/ai_observability/skills/exploring-llm-clusters/SKILL.md
index cf00de303624..f40968793865 100644
--- a/products/ai_observability/skills/exploring-llm-clusters/SKILL.md
+++ b/products/ai_observability/skills/exploring-llm-clusters/SKILL.md
@@ -306,6 +306,7 @@ For the full `$ai_*` event schema, load the `exploring-llm-traces` skill.
2. Calculate error rate per cluster: `items_with_errors / total_items`
3. Focus on clusters with high error rates
4. Drill into errored traces to find root causes
+5. If a cluster shows a failure that keeps recurring, offer to catch it with an online evaluation scoped to that cluster (`creating-online-evaluations`)
### "How do clusters compare across runs?"
diff --git a/products/ai_observability/skills/exploring-llm-traces/SKILL.md b/products/ai_observability/skills/exploring-llm-traces/SKILL.md
index 66c943cf7028..4703bd7b75a4 100644
--- a/products/ai_observability/skills/exploring-llm-traces/SKILL.md
+++ b/products/ai_observability/skills/exploring-llm-traces/SKILL.md
@@ -354,4 +354,5 @@ results (array for list, object for single trace)
- `$ai_input` / `$ai_output_choices` on generations contain the full LLM conversation — can be megabytes; when the result is persisted to a file, use the parsing scripts
- In raw SQL, heavy content (`$ai_input` / `$ai_output` / `$ai_output_choices` / `$ai_input_state` / `$ai_output_state` / `$ai_tools`) lives only on the `posthog.ai_events` table, not `events.properties` — see the [event reference](./references/events-and-properties.md) for the column mapping and trace-id-anchored query patterns
- Use `filterTestAccounts: true` to exclude internal/test traffic when searching
+- When an investigation turns up a failure that will happen again, not a one-off, offer to catch it with an online evaluation. Load `creating-online-evaluations` to set it up, so new traces get scored for it without anyone digging
- `$ai_trace` events are NOT in the `events` array — their data is surfaced via trace-level `inputState`, `outputState`, and `traceName`
diff --git a/services/mcp/schema/generated-tool-definitions.json b/services/mcp/schema/generated-tool-definitions.json
index 2e316d386b3b..787ceffea1b3 100644
--- a/services/mcp/schema/generated-tool-definitions.json
+++ b/services/mcp/schema/generated-tool-definitions.json
@@ -11907,7 +11907,7 @@
"system_prompt_hint": "List persons in a lifecycle bucket (new/returning/resurrecting/dormant) for a given day"
},
"query-llm-trace": {
- "description": "Fetch a single LLM trace by its trace ID for deep inspection. Returns the trace and every nested event, with model parameters, costs, tool calls, and errors. Use after finding a trace via `query-llm-traces-list` to inspect the complete event tree.\n\nBy default the response returns the retained event properties, subject to response size limits. Set `detail` to `\"summary\"` for metadata alone when browsing a trace, with no prompts or outputs in it.\n\nUse cases:\n\n- Inspect the full input/output of each generation in a trace\n- Debug a specific error trace found in the list\n- Examine the agent's decision-making flow across spans\n- Review tool calls and their results within a trace\n- Analyze token usage and costs per generation\n\nCRITICAL: This tool requires a `traceId`. Get the trace ID from `query-llm-traces-list` results first.\n\n# Response shape\n\nThe response contains a single trace in JSON format with:\n\n- `id` — the trace ID\n- `traceName` — name of the trace (if set via SDK)\n- `createdAt` — timestamp of the first event in the trace\n- `distinctId` — the person's distinct ID\n- `aiSessionId` — session ID grouping related traces (e.g., a conversation)\n- `totalLatency` — total latency in seconds\n- `inputTokens` / `outputTokens` — token counts across all generations\n- `inputCost` / `outputCost` / `totalCost` — costs in USD\n- `inputState` / `outputState` — JSON input/output state from the root `$ai_trace` event (e.g., conversation messages)\n- `events` — **all** child events in the trace at every nesting depth (not just direct children), subject to response size limits. Each event has `properties`, holding the retained properties in full by default and metadata only under `detail: \"summary\"`.\n\nUnlike `query-llm-traces-list`, this tool does NOT return `errorCount`, `isSupportTrace`, or `tools` — those are summary fields on the list tool only.\n\n# Event types and their properties\n\nEach event in `events` has an `event` field indicating its type. Key properties vary by type:\n\n- **`$ai_generation`** / **`$ai_embedding`** — an LLM or embedding API call. Properties include `$ai_input` (input prompt JSON), `$ai_output_choices` (output message JSON), `$ai_model`, `$ai_provider`, `$ai_latency`, `$ai_input_tokens`, `$ai_output_tokens`, `$ai_input_cost_usd`, `$ai_output_cost_usd`, `$ai_total_cost_usd`, `$ai_tools_called`, `$ai_is_error`, `$ai_error`.\n- **`$ai_span`** — a unit of work within a trace (e.g., a retrieval step, tool execution). Properties include `$ai_input_state`, `$ai_output_state`, `$ai_latency`, `$ai_span_name`, `$ai_parent_id`.\n- **`$ai_metric`** — a named evaluation metric. Properties include `$ai_metric_name`, `$ai_metric_value`.\n- **`$ai_feedback`** — user-provided feedback. Properties include `$ai_feedback_text`.\n\nNote: `$ai_trace` events are NOT included in the `events` array — their data is surfaced via the trace-level `inputState`, `outputState`, and `traceName` fields.\n\n# Tree structure (IDs and parent-child relationships)\n\nEvents in a trace form a tree. Each event carries three IDs that define its position:\n\n- `$ai_trace_id` — present on every event, identifies which trace it belongs to (same as the trace's `id`)\n- `$ai_span_id` (or `$ai_generation_id` for generations) — the event's own unique identifier\n- `$ai_parent_id` — points to the parent event's `$ai_span_id`\n\nTo reconstruct the tree:\n\n1. Events where `$ai_parent_id` equals `$ai_trace_id` are **root-level children** of the trace\n2. Other events are children of the event whose `$ai_span_id` matches their `$ai_parent_id`\n3. Group events by `$ai_parent_id` and walk from root children downward\n\nGenerations (`$ai_generation`) and embeddings (`$ai_embedding`) are always leaf nodes. Spans (`$ai_span`) can have children.\n\n# Examples\n\n## Fetch a trace by ID\n\n```json\n{\n \"kind\": \"TraceQuery\",\n \"traceId\": \"c9222e05-8708-41b8-98ea-d4a21849e761\"\n}\n```\n\n## Fetch with a date range hint\n\nIf the trace is old, provide a date range to help the query find it efficiently:\n\n```json\n{\n \"kind\": \"TraceQuery\",\n \"traceId\": \"c9222e05-8708-41b8-98ea-d4a21849e761\",\n \"dateRange\": { \"date_from\": \"-30d\" }\n}\n```\n\n# Detail level\n\n`detail` controls how much of each event you get back.\n\n- `\"full\"` (default) returns every retained property in full, bounded by the response size limit below.\n- `\"summary\"` opts into trace fields, plus each event's `id`, `createdAt`, `event` type, and its metadata: tree position (`$ai_trace_id`, `$ai_span_id`, `$ai_generation_id`, `$ai_parent_id`, `$ai_span_name`), the model (`$ai_model`, `$ai_provider`), timing (`$ai_latency`, `$ai_time_to_first_token`), the whole spend breakdown (every token count, per-token price, and per-modality cost), tool calls, metric and score values, and failure status (`$ai_is_error`, `$ai_http_status`, `$ai_error_type`, `$ai_status`, `$ai_stop_reason`). Everything else is left out, not shortened: their names are listed in `_summaryOmittedKeys` beside the bag. A summarized trace carries `_detail: { \"mode\": \"summary\" }`.\n- A summary carries no conversation content. Prompts, outputs, span states, `inputState` / `outputState`, `$ai_error`, and `$ai_feedback_text` all need `detail: \"full\"`. Use `$ai_is_error` and `$ai_http_status` to find the failed events in a summary, then read their messages at full detail. The trace and span names (`traceName`, `$ai_span_name`) do come back, and an SDK can write anything into those.\n\nFor a cost or latency survey, request `detail: \"summary\"`. Find the events that matter from their metadata, then re-run with `detail: \"full\"` when you need the text. Keep relevant date and property filters when requesting full detail.\n\n# Withheld properties\n\nOnly `$ai_*` properties PostHog's taxonomy defines, plus the ones first-party code writes without describing (`$ai_generation_id`, `$ai_cache_read_cost_usd`, `$ai_cache_creation_cost_usd`, `$ai_effort`) and `$session_id`, `$lib`, and `$lib_version`, reach you. Every other event property, and every person property, is withheld whichever `detail` you ask for, and its name is listed in `_redactedKeys` beside the bag. `$ai_base_url` and `$ai_request_url` arrive as the origin and path only, without the query string a provider key often sits in. A value that is not an `http` or `https` URL is withheld instead, because there is no endpoint to keep.\n\nA withheld property is unchanged in PostHog: it still works as a filter here, and you can read its value in the PostHog UI or with `execute-sql`.\n\n# Response size\n\nTo protect the agent's context window, very large traces are compacted before they reach you. Long string values are truncated (with a `… [truncated N chars]` marker), oversized arrays and objects have their tail members dropped (`… [N more items omitted]` / an `_omittedKeys` count), and if a trace is still over the size limit, trailing events are dropped and a `_truncated` object reports how many events were omitted. When you see any of these markers, open the trace in PostHog for the full, untruncated data, or narrow the query to the specific events you need. The underlying trace data is never altered, only this response is bounded. Both detail modes limit the complete response, including echoed filters and warnings, which can also be shortened with omission markers. Neither mode guarantees that every event or property fits. When a full-detail read drops events, re-run with `detail: \"summary\"` for metadata alone. Summary responses can also omit events.\n\n# Reminders\n\n- Always get the `traceId` from `query-llm-traces-list` results — do not guess or fabricate trace IDs.\n- If no date range is provided, the default lookback window is used. For older traces, provide an explicit `dateRange`.\n- The `events` array contains ALL events in the trace (including deeply nested ones), making this suitable for full tree reconstruction.\n- Use `query-llm-traces-list` first to find traces, then this tool to inspect a specific one.\n- Only ask for `detail: \"full\"` on a trace you have already narrowed down. A full-detail read of an unremarkable trace spends context you will need later.",
+ "description": "Fetch a single LLM trace by its trace ID for deep inspection. Returns the trace and every nested event, with model parameters, costs, tool calls, and errors. Use after finding a trace via `query-llm-traces-list` to inspect the complete event tree.\n\nBy default the response returns the retained event properties, subject to response size limits. Set `detail` to `\"summary\"` for metadata alone when browsing a trace, with no prompts or outputs in it.\n\nUse cases:\n\n- Inspect the full input/output of each generation in a trace\n- Debug a specific error trace found in the list\n- Examine the agent's decision-making flow across spans\n- Review tool calls and their results within a trace\n- Analyze token usage and costs per generation\n\nCRITICAL: This tool requires a `traceId`. Get the trace ID from `query-llm-traces-list` results first.\n\n# Response shape\n\nThe response contains a single trace in JSON format with:\n\n- `id` — the trace ID\n- `traceName` — name of the trace (if set via SDK)\n- `createdAt` — timestamp of the first event in the trace\n- `distinctId` — the person's distinct ID\n- `aiSessionId` — session ID grouping related traces (e.g., a conversation)\n- `totalLatency` — total latency in seconds\n- `inputTokens` / `outputTokens` — token counts across all generations\n- `inputCost` / `outputCost` / `totalCost` — costs in USD\n- `inputState` / `outputState` — JSON input/output state from the root `$ai_trace` event (e.g., conversation messages)\n- `events` — **all** child events in the trace at every nesting depth (not just direct children), subject to response size limits. Each event has `properties`, holding the retained properties in full by default and metadata only under `detail: \"summary\"`.\n\nUnlike `query-llm-traces-list`, this tool does NOT return `errorCount`, `isSupportTrace`, or `tools` — those are summary fields on the list tool only.\n\n# Event types and their properties\n\nEach event in `events` has an `event` field indicating its type. Key properties vary by type:\n\n- **`$ai_generation`** / **`$ai_embedding`** — an LLM or embedding API call. Properties include `$ai_input` (input prompt JSON), `$ai_output_choices` (output message JSON), `$ai_model`, `$ai_provider`, `$ai_latency`, `$ai_input_tokens`, `$ai_output_tokens`, `$ai_input_cost_usd`, `$ai_output_cost_usd`, `$ai_total_cost_usd`, `$ai_tools_called`, `$ai_is_error`, `$ai_error`.\n- **`$ai_span`** — a unit of work within a trace (e.g., a retrieval step, tool execution). Properties include `$ai_input_state`, `$ai_output_state`, `$ai_latency`, `$ai_span_name`, `$ai_parent_id`.\n- **`$ai_metric`** — a named evaluation metric. Properties include `$ai_metric_name`, `$ai_metric_value`.\n- **`$ai_feedback`** — user-provided feedback. Properties include `$ai_feedback_text`.\n\nNote: `$ai_trace` events are NOT included in the `events` array — their data is surfaced via the trace-level `inputState`, `outputState`, and `traceName` fields.\n\n# Tree structure (IDs and parent-child relationships)\n\nEvents in a trace form a tree. Each event carries three IDs that define its position:\n\n- `$ai_trace_id` — present on every event, identifies which trace it belongs to (same as the trace's `id`)\n- `$ai_span_id` (or `$ai_generation_id` for generations) — the event's own unique identifier\n- `$ai_parent_id` — points to the parent event's `$ai_span_id`\n\nTo reconstruct the tree:\n\n1. Events where `$ai_parent_id` equals `$ai_trace_id` are **root-level children** of the trace\n2. Other events are children of the event whose `$ai_span_id` matches their `$ai_parent_id`\n3. Group events by `$ai_parent_id` and walk from root children downward\n\nGenerations (`$ai_generation`) and embeddings (`$ai_embedding`) are always leaf nodes. Spans (`$ai_span`) can have children.\n\n# Examples\n\n## Fetch a trace by ID\n\n```json\n{\n \"kind\": \"TraceQuery\",\n \"traceId\": \"c9222e05-8708-41b8-98ea-d4a21849e761\"\n}\n```\n\n## Fetch with a date range hint\n\nIf the trace is old, provide a date range to help the query find it efficiently:\n\n```json\n{\n \"kind\": \"TraceQuery\",\n \"traceId\": \"c9222e05-8708-41b8-98ea-d4a21849e761\",\n \"dateRange\": { \"date_from\": \"-30d\" }\n}\n```\n\n# Detail level\n\n`detail` controls how much of each event you get back.\n\n- `\"full\"` (default) returns every retained property in full, bounded by the response size limit below.\n- `\"summary\"` opts into trace fields, plus each event's `id`, `createdAt`, `event` type, and its metadata: tree position (`$ai_trace_id`, `$ai_span_id`, `$ai_generation_id`, `$ai_parent_id`, `$ai_span_name`), the model (`$ai_model`, `$ai_provider`), timing (`$ai_latency`, `$ai_time_to_first_token`), the whole spend breakdown (every token count, per-token price, and per-modality cost), tool calls, metric and score values, and failure status (`$ai_is_error`, `$ai_http_status`, `$ai_error_type`, `$ai_status`, `$ai_stop_reason`). Everything else is left out, not shortened: their names are listed in `_summaryOmittedKeys` beside the bag. A summarized trace carries `_detail: { \"mode\": \"summary\" }`.\n- A summary carries no conversation content. Prompts, outputs, span states, `inputState` / `outputState`, `$ai_error`, and `$ai_feedback_text` all need `detail: \"full\"`. Use `$ai_is_error` and `$ai_http_status` to find the failed events in a summary, then read their messages at full detail. The trace and span names (`traceName`, `$ai_span_name`) do come back, and an SDK can write anything into those.\n\nFor a cost or latency survey, request `detail: \"summary\"`. Find the events that matter from their metadata, then re-run with `detail: \"full\"` when you need the text. Keep relevant date and property filters when requesting full detail.\n\n# Withheld properties\n\nOnly `$ai_*` properties PostHog's taxonomy defines, plus the ones first-party code writes without describing (`$ai_generation_id`, `$ai_cache_read_cost_usd`, `$ai_cache_creation_cost_usd`, `$ai_effort`) and `$session_id`, `$lib`, and `$lib_version`, reach you. Every other event property, and every person property, is withheld whichever `detail` you ask for, and its name is listed in `_redactedKeys` beside the bag. `$ai_base_url` and `$ai_request_url` arrive as the origin and path only, without the query string a provider key often sits in. A value that is not an `http` or `https` URL is withheld instead, because there is no endpoint to keep.\n\nA withheld property is unchanged in PostHog: it still works as a filter here, and you can read its value in the PostHog UI or with `execute-sql`.\n\n# Response size\n\nTo protect the agent's context window, very large traces are compacted before they reach you. Long string values are truncated (with a `… [truncated N chars]` marker), oversized arrays and objects have their tail members dropped (`… [N more items omitted]` / an `_omittedKeys` count), and if a trace is still over the size limit, trailing events are dropped and a `_truncated` object reports how many events were omitted. When you see any of these markers, open the trace in PostHog for the full, untruncated data, or narrow the query to the specific events you need. The underlying trace data is never altered, only this response is bounded. Both detail modes limit the complete response, including echoed filters and warnings, which can also be shortened with omission markers. Neither mode guarantees that every event or property fits. When a full-detail read drops events, re-run with `detail: \"summary\"` for metadata alone. Summary responses can also omit events.\n\n# Reminders\n\n- Always get the `traceId` from `query-llm-traces-list` results — do not guess or fabricate trace IDs.\n- If no date range is provided, the default lookback window is used. For older traces, provide an explicit `dateRange`.\n- The `events` array contains ALL events in the trace (including deeply nested ones), making this suitable for full tree reconstruction.\n- Use `query-llm-traces-list` first to find traces, then this tool to inspect a specific one.\n- Only ask for `detail: \"full\"` on a trace you have already narrowed down. A full-detail read of an unremarkable trace spends context you will need later.\n- When the traces show a failure that keeps happening, offer to score new traces for it automatically with an online evaluation. Load the `creating-online-evaluations` skill to set one up.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "Get LLM trace",
@@ -11923,7 +11923,7 @@
"system_prompt_hint": "Single LLM/AI trace deep-dive"
},
"query-llm-traces-list": {
- "description": "List LLM traces to inspect AI/LLM usage across your application. Returns traces with their events, latency, token usage, costs, errors, and other metadata. Use this tool for AI observability — debugging slow generations, investigating errors, analyzing token spend, and auditing LLM behavior.\n\nSet `detail: \"summary\"` for event metadata without any prompts or outputs when picking candidate traces, then read the one you pick with `query-llm-trace`. Omitting `detail` returns the retained properties in full, subject to size limits.\n\nUse 'read-data-schema' to discover available event properties for filtering (e.g. `$ai_model`, `$ai_provider`).\n\nExamples of use cases include:\n\n- How much are we spending on LLM tokens per day?\n- Which LLM generations are the slowest?\n- Are there any traces with errors in the last 24 hours?\n- What models are being used and how do their costs compare?\n- Show me traces for a specific user to debug their experience.\n- Are there any traces with unusually high token usage?\n\nCRITICAL: Be minimalist. Only include filters and settings that are essential to answer the user's specific question. Default settings are usually sufficient unless the user explicitly requests customization.\n\n# Data narrowing\n\n## Property filters\n\nUse property filters to narrow results. Only include property filters when they are essential to directly answer the user's question. Avoid adding them if the question can be addressed without additional segmentation and always use the minimum set of property filters needed.\n\nIMPORTANT: Do not check if a property is set unless the user explicitly asks for it.\n\nWhen using a property filter, you should:\n\n- **Prioritize properties directly related to the context or objective of the user's query.** Common AI properties include `$ai_model`, `$ai_provider`, `$ai_trace_id`, `$ai_session_id`, `$ai_latency`, `$ai_input_tokens`, `$ai_output_tokens`, `$ai_total_cost_usd`, `$ai_is_error`, `$ai_http_status`, `$ai_span_name`.\n- **Note:** `$ai_is_error` and `$ai_error` are valid filter properties but may not appear via `read-data-schema`. Use `$ai_is_error` with operator `exact` and value `[\"true\"]` to find error traces, or use `$ai_error` with `is set` to find traces with error messages.\n- **Ensure that you find both the property group and name.** Property groups should be one of the following: event, person, session, group.\n- After selecting a property, **validate that the property value accurately reflects the intended criteria**.\n- **Find the suitable operator for type** (e.g., `contains`, `is set`).\n- If the operator requires a value, use the `read-data-schema` tool to find the property values.\n\nInfer the property groups from the user's request. If your first guess doesn't yield any results, try to adjust the property group.\n\nSupported operators for the String type are:\n\n- equals (exact)\n- doesn't equal (is_not)\n- contains (icontains)\n- doesn't contain (not_icontains)\n- matches regex (regex)\n- doesn't match regex (not_regex)\n- is set\n- is not set\n\nSupported operators for the Numeric type are:\n\n- equals (exact)\n- doesn't equal (is_not)\n- greater than (gt)\n- less than (lt)\n- is set\n- is not set\n\nSupported operators for the DateTime type are:\n\n- equals (is_date_exact)\n- doesn't equal (is_not for existence check)\n- before (is_date_before)\n- after (is_date_after)\n- is set\n- is not set\n\nSupported operators for the Boolean type are:\n\n- equals\n- doesn't equal\n- is set\n- is not set\n\nAll operators take a single value except for `equals` and `doesn't equal` which can take one or more values (as an array).\n\n## Time period\n\nYou should not filter events by time using property filters. Instead, use the `dateRange` field. If the question doesn't mention time, use last 7 days as a default time period.\n\n# Traces guidelines\n\nThis is a listing tool, not a visualization/insight tool. It returns a paginated list of LLM traces — it does NOT support series, breakdowns, math aggregations, or chart types.\n\n## Response shape\n\nEach trace in the results contains:\n\n- `id` — unique trace ID\n- `traceName` — name of the trace (if set via SDK)\n- `createdAt` — timestamp of the first event in the trace\n- `distinctId` — the person's distinct ID\n- `aiSessionId` — session ID grouping related traces (e.g., a conversation)\n- `totalLatency` — total latency in seconds\n- `inputTokens` / `outputTokens` — token counts across all generations in the trace\n- `inputCost` / `outputCost` / `totalCost` — costs in USD\n- `inputState` / `outputState` — JSON input/output state of the trace (e.g., conversation messages), from the `$ai_trace` event\n- `errorCount` — number of errors in the trace\n- `isSupportTrace` — whether the trace was from a support impersonation session\n- `tools` — list of tool names called during the trace\n- `events` — list of direct child events (generations, metrics, feedback). Each event's `properties` contains the retained event data, returned in full by default and reduced to metadata under `detail: \"summary\"`, subject to response size limits. See \"Event types and their properties\" below.\n\n## Event types and their properties\n\nEach event in `events` has an `event` field indicating its type. The key properties vary by type:\n\n- **`$ai_generation`** / **`$ai_embedding`** — an LLM or embedding API call. Properties include `$ai_input` (input prompt JSON), `$ai_output_choices` (output message JSON), `$ai_model`, `$ai_provider`, `$ai_latency`, `$ai_input_tokens`, `$ai_output_tokens`, `$ai_input_cost_usd`, `$ai_output_cost_usd`, `$ai_total_cost_usd`, `$ai_tools_called`, `$ai_is_error`, `$ai_error`.\n- **`$ai_span`** — a unit of work within a trace (e.g., a retrieval step, a tool execution). Properties include `$ai_input_state`, `$ai_output_state`, `$ai_latency`, `$ai_span_name`, `$ai_parent_id`.\n- **`$ai_trace`** — the root trace event. Properties include `$ai_input_state` (e.g., conversation messages sent), `$ai_output_state` (e.g., final response), `$ai_span_name`.\n- **`$ai_metric`** — a named evaluation metric. Properties include `$ai_metric_name`, `$ai_metric_value`.\n- **`$ai_feedback`** — user-provided feedback. Properties include `$ai_feedback_text`.\n\nAll event types share `$ai_trace_id`, `$ai_span_id`, and `$ai_parent_id` for tree structure (see below).\n\n## Tree structure (IDs and parent-child relationships)\n\nEvents in a trace form a tree. Each event carries three IDs that define its position:\n\n- `$ai_trace_id` — present on every event, identifies which trace it belongs to (same as the trace's `id`)\n- `$ai_span_id` (or `$ai_generation_id` for generations) — the event's own unique identifier\n- `$ai_parent_id` — points to the parent event's `$ai_span_id`\n\nTo reconstruct the tree:\n\n1. Events where `$ai_parent_id` equals `$ai_trace_id` are **root-level children** of the trace\n2. Other events are children of the event whose `$ai_span_id` matches their `$ai_parent_id`\n3. Group events by `$ai_parent_id` and walk from root children downward\n\nGenerations (`$ai_generation`) and embeddings (`$ai_embedding`) are always leaf nodes. Spans (`$ai_span`) can have children.\n\n**Important:** This list tool only returns **direct children** of the trace (events where `$ai_parent_id` = trace ID) plus all `$ai_metric` and `$ai_feedback` events — NOT deeply nested events. For the full event tree with all nested children, use `query-llm-trace` with the trace's `id`.\n\n## Withheld properties\n\nOnly `$ai_*` properties PostHog's taxonomy defines, plus the ones first-party code writes without describing (`$ai_generation_id`, `$ai_cache_read_cost_usd`, `$ai_cache_creation_cost_usd`, `$ai_effort`) and `$session_id`, `$lib`, and `$lib_version`, reach you. Every other event property, and every person property, is withheld whichever `detail` you ask for, and its name is listed in `_redactedKeys` beside the bag. `$ai_base_url` and `$ai_request_url` arrive as the origin and path only, without the query string a provider key often sits in. A value that is not an `http` or `https` URL is withheld instead, because there is no endpoint to keep.\n\nA withheld property is unchanged in PostHog: it still works as a filter here, and you can read its value in the PostHog UI or with `execute-sql`.\n\n## Detail level\n\n`detail` controls how much of each event you get back.\n\n- `\"full\"` (default) returns every retained property in full, subject to response size limits.\n- `\"summary\"` opts into trace and event metadata only. It carries no conversation content: prompts, outputs, span states, `$ai_error`, and `$ai_feedback_text` are left out, and their names are listed in `_summaryOmittedKeys` beside the bag. The trace and span names (`traceName`, `$ai_span_name`) do come back, and an SDK can write anything into those. Use `$ai_is_error` and `$ai_http_status` to find the failed events, then read their messages with `detail: \"full\"`. A summarized trace carries `_detail: { \"mode\": \"summary\" }`.\n\nRequest `detail: \"summary\"` when finding candidate traces from their metadata; read the one you picked with `query-llm-trace` and `detail: \"full\"` when you need its content.\n\n## Response size\n\nTo protect the agent's context window, the response is bounded before it reaches you. Long string values are truncated (with a `… [truncated N chars]` marker) and oversized arrays/objects have their tail members dropped. If the combined list is still too large, trailing traces are dropped and a final `{ \"_truncated\": { \"omittedTraces\": N, ... } }` entry reports how many. When you see these markers, narrow the query (shorter date range, filters, or a smaller `limit`) or fetch a specific trace with `query-llm-trace`. The underlying data is never altered, only this response is bounded.\n\n## Pagination\n\nUse `limit` and `offset` for pagination. The default limit is 100. The response includes a `hasMore` field indicating whether more results are available.\n\n## Filtering\n\n- `filterTestAccounts` — exclude internal/test users\n- `filterSupportTraces` — exclude support impersonation traces\n- `personId` — filter by a specific person UUID\n- `groupKey` + `groupTypeIndex` — filter by a specific group\n- `randomOrder` — use random ordering instead of newest-first (useful for representative sampling)\n\n# Examples\n\n## Recent traces with errors\n\n```json\n{\n \"kind\": \"TracesQuery\",\n \"dateRange\": { \"date_from\": \"-7d\" },\n \"filterTestAccounts\": true,\n \"properties\": [{ \"key\": \"$ai_is_error\", \"operator\": \"exact\", \"type\": \"event\", \"value\": [\"true\"] }],\n \"limit\": 50\n}\n```\n\n## Traces for a specific model\n\n```json\n{\n \"kind\": \"TracesQuery\",\n \"dateRange\": { \"date_from\": \"-7d\" },\n \"filterTestAccounts\": true,\n \"properties\": [{ \"key\": \"$ai_model\", \"operator\": \"exact\", \"type\": \"event\", \"value\": [\"gpt-4o\"] }]\n}\n```\n\n## Traces for a specific person\n\n```json\n{\n \"kind\": \"TracesQuery\",\n \"dateRange\": { \"date_from\": \"-30d\" },\n \"personId\": \"01234567-89ab-cdef-0123-456789abcdef\",\n \"filterTestAccounts\": true\n}\n```\n\n## Random sample of traces (avoids recency bias)\n\n```json\n{\n \"kind\": \"TracesQuery\",\n \"dateRange\": { \"date_from\": \"-30d\" },\n \"filterTestAccounts\": true,\n \"randomOrder\": true,\n \"limit\": 20\n}\n```\n\n# Reminders\n\n- Ensure that any properties included are directly relevant to the context and objectives of the user's question. Avoid unnecessary or unrelated details.\n- Avoid overcomplicating the response with excessive property filters. Focus on the simplest solution.\n- This tool returns raw trace data — it does not aggregate or visualize. For aggregated LLM metrics over time (e.g. total token usage per day), use `query-trends` with AI events like `$ai_generation` instead.\n- Use `filterTestAccounts: true` by default to exclude internal users unless the user asks otherwise.\n- The default time range is last 7 days. LLM trace data tends to be recent, so shorter ranges are usually appropriate.\n- For deep inspection of a single trace (full event tree with all nested children and complete properties), use `query-llm-trace` with the trace's `id`.",
+ "description": "List LLM traces to inspect AI/LLM usage across your application. Returns traces with their events, latency, token usage, costs, errors, and other metadata. Use this tool for AI observability — debugging slow generations, investigating errors, analyzing token spend, and auditing LLM behavior.\n\nSet `detail: \"summary\"` for event metadata without any prompts or outputs when picking candidate traces, then read the one you pick with `query-llm-trace`. Omitting `detail` returns the retained properties in full, subject to size limits.\n\nUse 'read-data-schema' to discover available event properties for filtering (e.g. `$ai_model`, `$ai_provider`).\n\nExamples of use cases include:\n\n- How much are we spending on LLM tokens per day?\n- Which LLM generations are the slowest?\n- Are there any traces with errors in the last 24 hours?\n- What models are being used and how do their costs compare?\n- Show me traces for a specific user to debug their experience.\n- Are there any traces with unusually high token usage?\n\nCRITICAL: Be minimalist. Only include filters and settings that are essential to answer the user's specific question. Default settings are usually sufficient unless the user explicitly requests customization.\n\n# Data narrowing\n\n## Property filters\n\nUse property filters to narrow results. Only include property filters when they are essential to directly answer the user's question. Avoid adding them if the question can be addressed without additional segmentation and always use the minimum set of property filters needed.\n\nIMPORTANT: Do not check if a property is set unless the user explicitly asks for it.\n\nWhen using a property filter, you should:\n\n- **Prioritize properties directly related to the context or objective of the user's query.** Common AI properties include `$ai_model`, `$ai_provider`, `$ai_trace_id`, `$ai_session_id`, `$ai_latency`, `$ai_input_tokens`, `$ai_output_tokens`, `$ai_total_cost_usd`, `$ai_is_error`, `$ai_http_status`, `$ai_span_name`.\n- **Note:** `$ai_is_error` and `$ai_error` are valid filter properties but may not appear via `read-data-schema`. Use `$ai_is_error` with operator `exact` and value `[\"true\"]` to find error traces, or use `$ai_error` with `is set` to find traces with error messages.\n- **Ensure that you find both the property group and name.** Property groups should be one of the following: event, person, session, group.\n- After selecting a property, **validate that the property value accurately reflects the intended criteria**.\n- **Find the suitable operator for type** (e.g., `contains`, `is set`).\n- If the operator requires a value, use the `read-data-schema` tool to find the property values.\n\nInfer the property groups from the user's request. If your first guess doesn't yield any results, try to adjust the property group.\n\nSupported operators for the String type are:\n\n- equals (exact)\n- doesn't equal (is_not)\n- contains (icontains)\n- doesn't contain (not_icontains)\n- matches regex (regex)\n- doesn't match regex (not_regex)\n- is set\n- is not set\n\nSupported operators for the Numeric type are:\n\n- equals (exact)\n- doesn't equal (is_not)\n- greater than (gt)\n- less than (lt)\n- is set\n- is not set\n\nSupported operators for the DateTime type are:\n\n- equals (is_date_exact)\n- doesn't equal (is_not for existence check)\n- before (is_date_before)\n- after (is_date_after)\n- is set\n- is not set\n\nSupported operators for the Boolean type are:\n\n- equals\n- doesn't equal\n- is set\n- is not set\n\nAll operators take a single value except for `equals` and `doesn't equal` which can take one or more values (as an array).\n\n## Time period\n\nYou should not filter events by time using property filters. Instead, use the `dateRange` field. If the question doesn't mention time, use last 7 days as a default time period.\n\n# Traces guidelines\n\nThis is a listing tool, not a visualization/insight tool. It returns a paginated list of LLM traces — it does NOT support series, breakdowns, math aggregations, or chart types.\n\n## Response shape\n\nEach trace in the results contains:\n\n- `id` — unique trace ID\n- `traceName` — name of the trace (if set via SDK)\n- `createdAt` — timestamp of the first event in the trace\n- `distinctId` — the person's distinct ID\n- `aiSessionId` — session ID grouping related traces (e.g., a conversation)\n- `totalLatency` — total latency in seconds\n- `inputTokens` / `outputTokens` — token counts across all generations in the trace\n- `inputCost` / `outputCost` / `totalCost` — costs in USD\n- `inputState` / `outputState` — JSON input/output state of the trace (e.g., conversation messages), from the `$ai_trace` event\n- `errorCount` — number of errors in the trace\n- `isSupportTrace` — whether the trace was from a support impersonation session\n- `tools` — list of tool names called during the trace\n- `events` — list of direct child events (generations, metrics, feedback). Each event's `properties` contains the retained event data, returned in full by default and reduced to metadata under `detail: \"summary\"`, subject to response size limits. See \"Event types and their properties\" below.\n\n## Event types and their properties\n\nEach event in `events` has an `event` field indicating its type. The key properties vary by type:\n\n- **`$ai_generation`** / **`$ai_embedding`** — an LLM or embedding API call. Properties include `$ai_input` (input prompt JSON), `$ai_output_choices` (output message JSON), `$ai_model`, `$ai_provider`, `$ai_latency`, `$ai_input_tokens`, `$ai_output_tokens`, `$ai_input_cost_usd`, `$ai_output_cost_usd`, `$ai_total_cost_usd`, `$ai_tools_called`, `$ai_is_error`, `$ai_error`.\n- **`$ai_span`** — a unit of work within a trace (e.g., a retrieval step, a tool execution). Properties include `$ai_input_state`, `$ai_output_state`, `$ai_latency`, `$ai_span_name`, `$ai_parent_id`.\n- **`$ai_trace`** — the root trace event. Properties include `$ai_input_state` (e.g., conversation messages sent), `$ai_output_state` (e.g., final response), `$ai_span_name`.\n- **`$ai_metric`** — a named evaluation metric. Properties include `$ai_metric_name`, `$ai_metric_value`.\n- **`$ai_feedback`** — user-provided feedback. Properties include `$ai_feedback_text`.\n\nAll event types share `$ai_trace_id`, `$ai_span_id`, and `$ai_parent_id` for tree structure (see below).\n\n## Tree structure (IDs and parent-child relationships)\n\nEvents in a trace form a tree. Each event carries three IDs that define its position:\n\n- `$ai_trace_id` — present on every event, identifies which trace it belongs to (same as the trace's `id`)\n- `$ai_span_id` (or `$ai_generation_id` for generations) — the event's own unique identifier\n- `$ai_parent_id` — points to the parent event's `$ai_span_id`\n\nTo reconstruct the tree:\n\n1. Events where `$ai_parent_id` equals `$ai_trace_id` are **root-level children** of the trace\n2. Other events are children of the event whose `$ai_span_id` matches their `$ai_parent_id`\n3. Group events by `$ai_parent_id` and walk from root children downward\n\nGenerations (`$ai_generation`) and embeddings (`$ai_embedding`) are always leaf nodes. Spans (`$ai_span`) can have children.\n\n**Important:** This list tool only returns **direct children** of the trace (events where `$ai_parent_id` = trace ID) plus all `$ai_metric` and `$ai_feedback` events — NOT deeply nested events. For the full event tree with all nested children, use `query-llm-trace` with the trace's `id`.\n\n## Withheld properties\n\nOnly `$ai_*` properties PostHog's taxonomy defines, plus the ones first-party code writes without describing (`$ai_generation_id`, `$ai_cache_read_cost_usd`, `$ai_cache_creation_cost_usd`, `$ai_effort`) and `$session_id`, `$lib`, and `$lib_version`, reach you. Every other event property, and every person property, is withheld whichever `detail` you ask for, and its name is listed in `_redactedKeys` beside the bag. `$ai_base_url` and `$ai_request_url` arrive as the origin and path only, without the query string a provider key often sits in. A value that is not an `http` or `https` URL is withheld instead, because there is no endpoint to keep.\n\nA withheld property is unchanged in PostHog: it still works as a filter here, and you can read its value in the PostHog UI or with `execute-sql`.\n\n## Detail level\n\n`detail` controls how much of each event you get back.\n\n- `\"full\"` (default) returns every retained property in full, subject to response size limits.\n- `\"summary\"` opts into trace and event metadata only. It carries no conversation content: prompts, outputs, span states, `$ai_error`, and `$ai_feedback_text` are left out, and their names are listed in `_summaryOmittedKeys` beside the bag. The trace and span names (`traceName`, `$ai_span_name`) do come back, and an SDK can write anything into those. Use `$ai_is_error` and `$ai_http_status` to find the failed events, then read their messages with `detail: \"full\"`. A summarized trace carries `_detail: { \"mode\": \"summary\" }`.\n\nRequest `detail: \"summary\"` when finding candidate traces from their metadata; read the one you picked with `query-llm-trace` and `detail: \"full\"` when you need its content.\n\n## Response size\n\nTo protect the agent's context window, the response is bounded before it reaches you. Long string values are truncated (with a `… [truncated N chars]` marker) and oversized arrays/objects have their tail members dropped. If the combined list is still too large, trailing traces are dropped and a final `{ \"_truncated\": { \"omittedTraces\": N, ... } }` entry reports how many. When you see these markers, narrow the query (shorter date range, filters, or a smaller `limit`) or fetch a specific trace with `query-llm-trace`. The underlying data is never altered, only this response is bounded.\n\n## Pagination\n\nUse `limit` and `offset` for pagination. The default limit is 100. The response includes a `hasMore` field indicating whether more results are available.\n\n## Filtering\n\n- `filterTestAccounts` — exclude internal/test users\n- `filterSupportTraces` — exclude support impersonation traces\n- `personId` — filter by a specific person UUID\n- `groupKey` + `groupTypeIndex` — filter by a specific group\n- `randomOrder` — use random ordering instead of newest-first (useful for representative sampling)\n\n# Examples\n\n## Recent traces with errors\n\n```json\n{\n \"kind\": \"TracesQuery\",\n \"dateRange\": { \"date_from\": \"-7d\" },\n \"filterTestAccounts\": true,\n \"properties\": [{ \"key\": \"$ai_is_error\", \"operator\": \"exact\", \"type\": \"event\", \"value\": [\"true\"] }],\n \"limit\": 50\n}\n```\n\n## Traces for a specific model\n\n```json\n{\n \"kind\": \"TracesQuery\",\n \"dateRange\": { \"date_from\": \"-7d\" },\n \"filterTestAccounts\": true,\n \"properties\": [{ \"key\": \"$ai_model\", \"operator\": \"exact\", \"type\": \"event\", \"value\": [\"gpt-4o\"] }]\n}\n```\n\n## Traces for a specific person\n\n```json\n{\n \"kind\": \"TracesQuery\",\n \"dateRange\": { \"date_from\": \"-30d\" },\n \"personId\": \"01234567-89ab-cdef-0123-456789abcdef\",\n \"filterTestAccounts\": true\n}\n```\n\n## Random sample of traces (avoids recency bias)\n\n```json\n{\n \"kind\": \"TracesQuery\",\n \"dateRange\": { \"date_from\": \"-30d\" },\n \"filterTestAccounts\": true,\n \"randomOrder\": true,\n \"limit\": 20\n}\n```\n\n# Reminders\n\n- Ensure that any properties included are directly relevant to the context and objectives of the user's question. Avoid unnecessary or unrelated details.\n- Avoid overcomplicating the response with excessive property filters. Focus on the simplest solution.\n- This tool returns raw trace data — it does not aggregate or visualize. For aggregated LLM metrics over time (e.g. total token usage per day), use `query-trends` with AI events like `$ai_generation` instead.\n- Use `filterTestAccounts: true` by default to exclude internal users unless the user asks otherwise.\n- The default time range is last 7 days. LLM trace data tends to be recent, so shorter ranges are usually appropriate.\n- For deep inspection of a single trace (full event tree with all nested children and complete properties), use `query-llm-trace` with the trace's `id`.\n- When the traces show a failure that keeps happening, offer to score new traces for it automatically with an online evaluation. Load the `creating-online-evaluations` skill to set one up.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "List LLM traces",
diff --git a/services/mcp/schema/tool-definitions-all.json b/services/mcp/schema/tool-definitions-all.json
index b16390fdf5ee..223d4ed24336 100644
--- a/services/mcp/schema/tool-definitions-all.json
+++ b/services/mcp/schema/tool-definitions-all.json
@@ -12593,7 +12593,7 @@
"system_prompt_hint": "List persons in a lifecycle bucket (new/returning/resurrecting/dormant) for a given day"
},
"query-llm-trace": {
- "description": "Fetch a single LLM trace by its trace ID for deep inspection. Returns the trace and every nested event, with model parameters, costs, tool calls, and errors. Use after finding a trace via `query-llm-traces-list` to inspect the complete event tree.\n\nBy default the response returns the retained event properties, subject to response size limits. Set `detail` to `\"summary\"` for metadata alone when browsing a trace, with no prompts or outputs in it.\n\nUse cases:\n\n- Inspect the full input/output of each generation in a trace\n- Debug a specific error trace found in the list\n- Examine the agent's decision-making flow across spans\n- Review tool calls and their results within a trace\n- Analyze token usage and costs per generation\n\nCRITICAL: This tool requires a `traceId`. Get the trace ID from `query-llm-traces-list` results first.\n\n# Response shape\n\nThe response contains a single trace in JSON format with:\n\n- `id` — the trace ID\n- `traceName` — name of the trace (if set via SDK)\n- `createdAt` — timestamp of the first event in the trace\n- `distinctId` — the person's distinct ID\n- `aiSessionId` — session ID grouping related traces (e.g., a conversation)\n- `totalLatency` — total latency in seconds\n- `inputTokens` / `outputTokens` — token counts across all generations\n- `inputCost` / `outputCost` / `totalCost` — costs in USD\n- `inputState` / `outputState` — JSON input/output state from the root `$ai_trace` event (e.g., conversation messages)\n- `events` — **all** child events in the trace at every nesting depth (not just direct children), subject to response size limits. Each event has `properties`, holding the retained properties in full by default and metadata only under `detail: \"summary\"`.\n\nUnlike `query-llm-traces-list`, this tool does NOT return `errorCount`, `isSupportTrace`, or `tools` — those are summary fields on the list tool only.\n\n# Event types and their properties\n\nEach event in `events` has an `event` field indicating its type. Key properties vary by type:\n\n- **`$ai_generation`** / **`$ai_embedding`** — an LLM or embedding API call. Properties include `$ai_input` (input prompt JSON), `$ai_output_choices` (output message JSON), `$ai_model`, `$ai_provider`, `$ai_latency`, `$ai_input_tokens`, `$ai_output_tokens`, `$ai_input_cost_usd`, `$ai_output_cost_usd`, `$ai_total_cost_usd`, `$ai_tools_called`, `$ai_is_error`, `$ai_error`.\n- **`$ai_span`** — a unit of work within a trace (e.g., a retrieval step, tool execution). Properties include `$ai_input_state`, `$ai_output_state`, `$ai_latency`, `$ai_span_name`, `$ai_parent_id`.\n- **`$ai_metric`** — a named evaluation metric. Properties include `$ai_metric_name`, `$ai_metric_value`.\n- **`$ai_feedback`** — user-provided feedback. Properties include `$ai_feedback_text`.\n\nNote: `$ai_trace` events are NOT included in the `events` array — their data is surfaced via the trace-level `inputState`, `outputState`, and `traceName` fields.\n\n# Tree structure (IDs and parent-child relationships)\n\nEvents in a trace form a tree. Each event carries three IDs that define its position:\n\n- `$ai_trace_id` — present on every event, identifies which trace it belongs to (same as the trace's `id`)\n- `$ai_span_id` (or `$ai_generation_id` for generations) — the event's own unique identifier\n- `$ai_parent_id` — points to the parent event's `$ai_span_id`\n\nTo reconstruct the tree:\n\n1. Events where `$ai_parent_id` equals `$ai_trace_id` are **root-level children** of the trace\n2. Other events are children of the event whose `$ai_span_id` matches their `$ai_parent_id`\n3. Group events by `$ai_parent_id` and walk from root children downward\n\nGenerations (`$ai_generation`) and embeddings (`$ai_embedding`) are always leaf nodes. Spans (`$ai_span`) can have children.\n\n# Examples\n\n## Fetch a trace by ID\n\n```json\n{\n \"kind\": \"TraceQuery\",\n \"traceId\": \"c9222e05-8708-41b8-98ea-d4a21849e761\"\n}\n```\n\n## Fetch with a date range hint\n\nIf the trace is old, provide a date range to help the query find it efficiently:\n\n```json\n{\n \"kind\": \"TraceQuery\",\n \"traceId\": \"c9222e05-8708-41b8-98ea-d4a21849e761\",\n \"dateRange\": { \"date_from\": \"-30d\" }\n}\n```\n\n# Detail level\n\n`detail` controls how much of each event you get back.\n\n- `\"full\"` (default) returns every retained property in full, bounded by the response size limit below.\n- `\"summary\"` opts into trace fields, plus each event's `id`, `createdAt`, `event` type, and its metadata: tree position (`$ai_trace_id`, `$ai_span_id`, `$ai_generation_id`, `$ai_parent_id`, `$ai_span_name`), the model (`$ai_model`, `$ai_provider`), timing (`$ai_latency`, `$ai_time_to_first_token`), the whole spend breakdown (every token count, per-token price, and per-modality cost), tool calls, metric and score values, and failure status (`$ai_is_error`, `$ai_http_status`, `$ai_error_type`, `$ai_status`, `$ai_stop_reason`). Everything else is left out, not shortened: their names are listed in `_summaryOmittedKeys` beside the bag. A summarized trace carries `_detail: { \"mode\": \"summary\" }`.\n- A summary carries no conversation content. Prompts, outputs, span states, `inputState` / `outputState`, `$ai_error`, and `$ai_feedback_text` all need `detail: \"full\"`. Use `$ai_is_error` and `$ai_http_status` to find the failed events in a summary, then read their messages at full detail. The trace and span names (`traceName`, `$ai_span_name`) do come back, and an SDK can write anything into those.\n\nFor a cost or latency survey, request `detail: \"summary\"`. Find the events that matter from their metadata, then re-run with `detail: \"full\"` when you need the text. Keep relevant date and property filters when requesting full detail.\n\n# Withheld properties\n\nOnly `$ai_*` properties PostHog's taxonomy defines, plus the ones first-party code writes without describing (`$ai_generation_id`, `$ai_cache_read_cost_usd`, `$ai_cache_creation_cost_usd`, `$ai_effort`) and `$session_id`, `$lib`, and `$lib_version`, reach you. Every other event property, and every person property, is withheld whichever `detail` you ask for, and its name is listed in `_redactedKeys` beside the bag. `$ai_base_url` and `$ai_request_url` arrive as the origin and path only, without the query string a provider key often sits in. A value that is not an `http` or `https` URL is withheld instead, because there is no endpoint to keep.\n\nA withheld property is unchanged in PostHog: it still works as a filter here, and you can read its value in the PostHog UI or with `execute-sql`.\n\n# Response size\n\nTo protect the agent's context window, very large traces are compacted before they reach you. Long string values are truncated (with a `… [truncated N chars]` marker), oversized arrays and objects have their tail members dropped (`… [N more items omitted]` / an `_omittedKeys` count), and if a trace is still over the size limit, trailing events are dropped and a `_truncated` object reports how many events were omitted. When you see any of these markers, open the trace in PostHog for the full, untruncated data, or narrow the query to the specific events you need. The underlying trace data is never altered, only this response is bounded. Both detail modes limit the complete response, including echoed filters and warnings, which can also be shortened with omission markers. Neither mode guarantees that every event or property fits. When a full-detail read drops events, re-run with `detail: \"summary\"` for metadata alone. Summary responses can also omit events.\n\n# Reminders\n\n- Always get the `traceId` from `query-llm-traces-list` results — do not guess or fabricate trace IDs.\n- If no date range is provided, the default lookback window is used. For older traces, provide an explicit `dateRange`.\n- The `events` array contains ALL events in the trace (including deeply nested ones), making this suitable for full tree reconstruction.\n- Use `query-llm-traces-list` first to find traces, then this tool to inspect a specific one.\n- Only ask for `detail: \"full\"` on a trace you have already narrowed down. A full-detail read of an unremarkable trace spends context you will need later.",
+ "description": "Fetch a single LLM trace by its trace ID for deep inspection. Returns the trace and every nested event, with model parameters, costs, tool calls, and errors. Use after finding a trace via `query-llm-traces-list` to inspect the complete event tree.\n\nBy default the response returns the retained event properties, subject to response size limits. Set `detail` to `\"summary\"` for metadata alone when browsing a trace, with no prompts or outputs in it.\n\nUse cases:\n\n- Inspect the full input/output of each generation in a trace\n- Debug a specific error trace found in the list\n- Examine the agent's decision-making flow across spans\n- Review tool calls and their results within a trace\n- Analyze token usage and costs per generation\n\nCRITICAL: This tool requires a `traceId`. Get the trace ID from `query-llm-traces-list` results first.\n\n# Response shape\n\nThe response contains a single trace in JSON format with:\n\n- `id` — the trace ID\n- `traceName` — name of the trace (if set via SDK)\n- `createdAt` — timestamp of the first event in the trace\n- `distinctId` — the person's distinct ID\n- `aiSessionId` — session ID grouping related traces (e.g., a conversation)\n- `totalLatency` — total latency in seconds\n- `inputTokens` / `outputTokens` — token counts across all generations\n- `inputCost` / `outputCost` / `totalCost` — costs in USD\n- `inputState` / `outputState` — JSON input/output state from the root `$ai_trace` event (e.g., conversation messages)\n- `events` — **all** child events in the trace at every nesting depth (not just direct children), subject to response size limits. Each event has `properties`, holding the retained properties in full by default and metadata only under `detail: \"summary\"`.\n\nUnlike `query-llm-traces-list`, this tool does NOT return `errorCount`, `isSupportTrace`, or `tools` — those are summary fields on the list tool only.\n\n# Event types and their properties\n\nEach event in `events` has an `event` field indicating its type. Key properties vary by type:\n\n- **`$ai_generation`** / **`$ai_embedding`** — an LLM or embedding API call. Properties include `$ai_input` (input prompt JSON), `$ai_output_choices` (output message JSON), `$ai_model`, `$ai_provider`, `$ai_latency`, `$ai_input_tokens`, `$ai_output_tokens`, `$ai_input_cost_usd`, `$ai_output_cost_usd`, `$ai_total_cost_usd`, `$ai_tools_called`, `$ai_is_error`, `$ai_error`.\n- **`$ai_span`** — a unit of work within a trace (e.g., a retrieval step, tool execution). Properties include `$ai_input_state`, `$ai_output_state`, `$ai_latency`, `$ai_span_name`, `$ai_parent_id`.\n- **`$ai_metric`** — a named evaluation metric. Properties include `$ai_metric_name`, `$ai_metric_value`.\n- **`$ai_feedback`** — user-provided feedback. Properties include `$ai_feedback_text`.\n\nNote: `$ai_trace` events are NOT included in the `events` array — their data is surfaced via the trace-level `inputState`, `outputState`, and `traceName` fields.\n\n# Tree structure (IDs and parent-child relationships)\n\nEvents in a trace form a tree. Each event carries three IDs that define its position:\n\n- `$ai_trace_id` — present on every event, identifies which trace it belongs to (same as the trace's `id`)\n- `$ai_span_id` (or `$ai_generation_id` for generations) — the event's own unique identifier\n- `$ai_parent_id` — points to the parent event's `$ai_span_id`\n\nTo reconstruct the tree:\n\n1. Events where `$ai_parent_id` equals `$ai_trace_id` are **root-level children** of the trace\n2. Other events are children of the event whose `$ai_span_id` matches their `$ai_parent_id`\n3. Group events by `$ai_parent_id` and walk from root children downward\n\nGenerations (`$ai_generation`) and embeddings (`$ai_embedding`) are always leaf nodes. Spans (`$ai_span`) can have children.\n\n# Examples\n\n## Fetch a trace by ID\n\n```json\n{\n \"kind\": \"TraceQuery\",\n \"traceId\": \"c9222e05-8708-41b8-98ea-d4a21849e761\"\n}\n```\n\n## Fetch with a date range hint\n\nIf the trace is old, provide a date range to help the query find it efficiently:\n\n```json\n{\n \"kind\": \"TraceQuery\",\n \"traceId\": \"c9222e05-8708-41b8-98ea-d4a21849e761\",\n \"dateRange\": { \"date_from\": \"-30d\" }\n}\n```\n\n# Detail level\n\n`detail` controls how much of each event you get back.\n\n- `\"full\"` (default) returns every retained property in full, bounded by the response size limit below.\n- `\"summary\"` opts into trace fields, plus each event's `id`, `createdAt`, `event` type, and its metadata: tree position (`$ai_trace_id`, `$ai_span_id`, `$ai_generation_id`, `$ai_parent_id`, `$ai_span_name`), the model (`$ai_model`, `$ai_provider`), timing (`$ai_latency`, `$ai_time_to_first_token`), the whole spend breakdown (every token count, per-token price, and per-modality cost), tool calls, metric and score values, and failure status (`$ai_is_error`, `$ai_http_status`, `$ai_error_type`, `$ai_status`, `$ai_stop_reason`). Everything else is left out, not shortened: their names are listed in `_summaryOmittedKeys` beside the bag. A summarized trace carries `_detail: { \"mode\": \"summary\" }`.\n- A summary carries no conversation content. Prompts, outputs, span states, `inputState` / `outputState`, `$ai_error`, and `$ai_feedback_text` all need `detail: \"full\"`. Use `$ai_is_error` and `$ai_http_status` to find the failed events in a summary, then read their messages at full detail. The trace and span names (`traceName`, `$ai_span_name`) do come back, and an SDK can write anything into those.\n\nFor a cost or latency survey, request `detail: \"summary\"`. Find the events that matter from their metadata, then re-run with `detail: \"full\"` when you need the text. Keep relevant date and property filters when requesting full detail.\n\n# Withheld properties\n\nOnly `$ai_*` properties PostHog's taxonomy defines, plus the ones first-party code writes without describing (`$ai_generation_id`, `$ai_cache_read_cost_usd`, `$ai_cache_creation_cost_usd`, `$ai_effort`) and `$session_id`, `$lib`, and `$lib_version`, reach you. Every other event property, and every person property, is withheld whichever `detail` you ask for, and its name is listed in `_redactedKeys` beside the bag. `$ai_base_url` and `$ai_request_url` arrive as the origin and path only, without the query string a provider key often sits in. A value that is not an `http` or `https` URL is withheld instead, because there is no endpoint to keep.\n\nA withheld property is unchanged in PostHog: it still works as a filter here, and you can read its value in the PostHog UI or with `execute-sql`.\n\n# Response size\n\nTo protect the agent's context window, very large traces are compacted before they reach you. Long string values are truncated (with a `… [truncated N chars]` marker), oversized arrays and objects have their tail members dropped (`… [N more items omitted]` / an `_omittedKeys` count), and if a trace is still over the size limit, trailing events are dropped and a `_truncated` object reports how many events were omitted. When you see any of these markers, open the trace in PostHog for the full, untruncated data, or narrow the query to the specific events you need. The underlying trace data is never altered, only this response is bounded. Both detail modes limit the complete response, including echoed filters and warnings, which can also be shortened with omission markers. Neither mode guarantees that every event or property fits. When a full-detail read drops events, re-run with `detail: \"summary\"` for metadata alone. Summary responses can also omit events.\n\n# Reminders\n\n- Always get the `traceId` from `query-llm-traces-list` results — do not guess or fabricate trace IDs.\n- If no date range is provided, the default lookback window is used. For older traces, provide an explicit `dateRange`.\n- The `events` array contains ALL events in the trace (including deeply nested ones), making this suitable for full tree reconstruction.\n- Use `query-llm-traces-list` first to find traces, then this tool to inspect a specific one.\n- Only ask for `detail: \"full\"` on a trace you have already narrowed down. A full-detail read of an unremarkable trace spends context you will need later.\n- When the traces show a failure that keeps happening, offer to score new traces for it automatically with an online evaluation. Load the `creating-online-evaluations` skill to set one up.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "Get LLM trace",
@@ -12609,7 +12609,7 @@
"system_prompt_hint": "Single LLM/AI trace deep-dive"
},
"query-llm-traces-list": {
- "description": "List LLM traces to inspect AI/LLM usage across your application. Returns traces with their events, latency, token usage, costs, errors, and other metadata. Use this tool for AI observability — debugging slow generations, investigating errors, analyzing token spend, and auditing LLM behavior.\n\nSet `detail: \"summary\"` for event metadata without any prompts or outputs when picking candidate traces, then read the one you pick with `query-llm-trace`. Omitting `detail` returns the retained properties in full, subject to size limits.\n\nUse 'read-data-schema' to discover available event properties for filtering (e.g. `$ai_model`, `$ai_provider`).\n\nExamples of use cases include:\n\n- How much are we spending on LLM tokens per day?\n- Which LLM generations are the slowest?\n- Are there any traces with errors in the last 24 hours?\n- What models are being used and how do their costs compare?\n- Show me traces for a specific user to debug their experience.\n- Are there any traces with unusually high token usage?\n\nCRITICAL: Be minimalist. Only include filters and settings that are essential to answer the user's specific question. Default settings are usually sufficient unless the user explicitly requests customization.\n\n# Data narrowing\n\n## Property filters\n\nUse property filters to narrow results. Only include property filters when they are essential to directly answer the user's question. Avoid adding them if the question can be addressed without additional segmentation and always use the minimum set of property filters needed.\n\nIMPORTANT: Do not check if a property is set unless the user explicitly asks for it.\n\nWhen using a property filter, you should:\n\n- **Prioritize properties directly related to the context or objective of the user's query.** Common AI properties include `$ai_model`, `$ai_provider`, `$ai_trace_id`, `$ai_session_id`, `$ai_latency`, `$ai_input_tokens`, `$ai_output_tokens`, `$ai_total_cost_usd`, `$ai_is_error`, `$ai_http_status`, `$ai_span_name`.\n- **Note:** `$ai_is_error` and `$ai_error` are valid filter properties but may not appear via `read-data-schema`. Use `$ai_is_error` with operator `exact` and value `[\"true\"]` to find error traces, or use `$ai_error` with `is set` to find traces with error messages.\n- **Ensure that you find both the property group and name.** Property groups should be one of the following: event, person, session, group.\n- After selecting a property, **validate that the property value accurately reflects the intended criteria**.\n- **Find the suitable operator for type** (e.g., `contains`, `is set`).\n- If the operator requires a value, use the `read-data-schema` tool to find the property values.\n\nInfer the property groups from the user's request. If your first guess doesn't yield any results, try to adjust the property group.\n\nSupported operators for the String type are:\n\n- equals (exact)\n- doesn't equal (is_not)\n- contains (icontains)\n- doesn't contain (not_icontains)\n- matches regex (regex)\n- doesn't match regex (not_regex)\n- is set\n- is not set\n\nSupported operators for the Numeric type are:\n\n- equals (exact)\n- doesn't equal (is_not)\n- greater than (gt)\n- less than (lt)\n- is set\n- is not set\n\nSupported operators for the DateTime type are:\n\n- equals (is_date_exact)\n- doesn't equal (is_not for existence check)\n- before (is_date_before)\n- after (is_date_after)\n- is set\n- is not set\n\nSupported operators for the Boolean type are:\n\n- equals\n- doesn't equal\n- is set\n- is not set\n\nAll operators take a single value except for `equals` and `doesn't equal` which can take one or more values (as an array).\n\n## Time period\n\nYou should not filter events by time using property filters. Instead, use the `dateRange` field. If the question doesn't mention time, use last 7 days as a default time period.\n\n# Traces guidelines\n\nThis is a listing tool, not a visualization/insight tool. It returns a paginated list of LLM traces — it does NOT support series, breakdowns, math aggregations, or chart types.\n\n## Response shape\n\nEach trace in the results contains:\n\n- `id` — unique trace ID\n- `traceName` — name of the trace (if set via SDK)\n- `createdAt` — timestamp of the first event in the trace\n- `distinctId` — the person's distinct ID\n- `aiSessionId` — session ID grouping related traces (e.g., a conversation)\n- `totalLatency` — total latency in seconds\n- `inputTokens` / `outputTokens` — token counts across all generations in the trace\n- `inputCost` / `outputCost` / `totalCost` — costs in USD\n- `inputState` / `outputState` — JSON input/output state of the trace (e.g., conversation messages), from the `$ai_trace` event\n- `errorCount` — number of errors in the trace\n- `isSupportTrace` — whether the trace was from a support impersonation session\n- `tools` — list of tool names called during the trace\n- `events` — list of direct child events (generations, metrics, feedback). Each event's `properties` contains the retained event data, returned in full by default and reduced to metadata under `detail: \"summary\"`, subject to response size limits. See \"Event types and their properties\" below.\n\n## Event types and their properties\n\nEach event in `events` has an `event` field indicating its type. The key properties vary by type:\n\n- **`$ai_generation`** / **`$ai_embedding`** — an LLM or embedding API call. Properties include `$ai_input` (input prompt JSON), `$ai_output_choices` (output message JSON), `$ai_model`, `$ai_provider`, `$ai_latency`, `$ai_input_tokens`, `$ai_output_tokens`, `$ai_input_cost_usd`, `$ai_output_cost_usd`, `$ai_total_cost_usd`, `$ai_tools_called`, `$ai_is_error`, `$ai_error`.\n- **`$ai_span`** — a unit of work within a trace (e.g., a retrieval step, a tool execution). Properties include `$ai_input_state`, `$ai_output_state`, `$ai_latency`, `$ai_span_name`, `$ai_parent_id`.\n- **`$ai_trace`** — the root trace event. Properties include `$ai_input_state` (e.g., conversation messages sent), `$ai_output_state` (e.g., final response), `$ai_span_name`.\n- **`$ai_metric`** — a named evaluation metric. Properties include `$ai_metric_name`, `$ai_metric_value`.\n- **`$ai_feedback`** — user-provided feedback. Properties include `$ai_feedback_text`.\n\nAll event types share `$ai_trace_id`, `$ai_span_id`, and `$ai_parent_id` for tree structure (see below).\n\n## Tree structure (IDs and parent-child relationships)\n\nEvents in a trace form a tree. Each event carries three IDs that define its position:\n\n- `$ai_trace_id` — present on every event, identifies which trace it belongs to (same as the trace's `id`)\n- `$ai_span_id` (or `$ai_generation_id` for generations) — the event's own unique identifier\n- `$ai_parent_id` — points to the parent event's `$ai_span_id`\n\nTo reconstruct the tree:\n\n1. Events where `$ai_parent_id` equals `$ai_trace_id` are **root-level children** of the trace\n2. Other events are children of the event whose `$ai_span_id` matches their `$ai_parent_id`\n3. Group events by `$ai_parent_id` and walk from root children downward\n\nGenerations (`$ai_generation`) and embeddings (`$ai_embedding`) are always leaf nodes. Spans (`$ai_span`) can have children.\n\n**Important:** This list tool only returns **direct children** of the trace (events where `$ai_parent_id` = trace ID) plus all `$ai_metric` and `$ai_feedback` events — NOT deeply nested events. For the full event tree with all nested children, use `query-llm-trace` with the trace's `id`.\n\n## Withheld properties\n\nOnly `$ai_*` properties PostHog's taxonomy defines, plus the ones first-party code writes without describing (`$ai_generation_id`, `$ai_cache_read_cost_usd`, `$ai_cache_creation_cost_usd`, `$ai_effort`) and `$session_id`, `$lib`, and `$lib_version`, reach you. Every other event property, and every person property, is withheld whichever `detail` you ask for, and its name is listed in `_redactedKeys` beside the bag. `$ai_base_url` and `$ai_request_url` arrive as the origin and path only, without the query string a provider key often sits in. A value that is not an `http` or `https` URL is withheld instead, because there is no endpoint to keep.\n\nA withheld property is unchanged in PostHog: it still works as a filter here, and you can read its value in the PostHog UI or with `execute-sql`.\n\n## Detail level\n\n`detail` controls how much of each event you get back.\n\n- `\"full\"` (default) returns every retained property in full, subject to response size limits.\n- `\"summary\"` opts into trace and event metadata only. It carries no conversation content: prompts, outputs, span states, `$ai_error`, and `$ai_feedback_text` are left out, and their names are listed in `_summaryOmittedKeys` beside the bag. The trace and span names (`traceName`, `$ai_span_name`) do come back, and an SDK can write anything into those. Use `$ai_is_error` and `$ai_http_status` to find the failed events, then read their messages with `detail: \"full\"`. A summarized trace carries `_detail: { \"mode\": \"summary\" }`.\n\nRequest `detail: \"summary\"` when finding candidate traces from their metadata; read the one you picked with `query-llm-trace` and `detail: \"full\"` when you need its content.\n\n## Response size\n\nTo protect the agent's context window, the response is bounded before it reaches you. Long string values are truncated (with a `… [truncated N chars]` marker) and oversized arrays/objects have their tail members dropped. If the combined list is still too large, trailing traces are dropped and a final `{ \"_truncated\": { \"omittedTraces\": N, ... } }` entry reports how many. When you see these markers, narrow the query (shorter date range, filters, or a smaller `limit`) or fetch a specific trace with `query-llm-trace`. The underlying data is never altered, only this response is bounded.\n\n## Pagination\n\nUse `limit` and `offset` for pagination. The default limit is 100. The response includes a `hasMore` field indicating whether more results are available.\n\n## Filtering\n\n- `filterTestAccounts` — exclude internal/test users\n- `filterSupportTraces` — exclude support impersonation traces\n- `personId` — filter by a specific person UUID\n- `groupKey` + `groupTypeIndex` — filter by a specific group\n- `randomOrder` — use random ordering instead of newest-first (useful for representative sampling)\n\n# Examples\n\n## Recent traces with errors\n\n```json\n{\n \"kind\": \"TracesQuery\",\n \"dateRange\": { \"date_from\": \"-7d\" },\n \"filterTestAccounts\": true,\n \"properties\": [{ \"key\": \"$ai_is_error\", \"operator\": \"exact\", \"type\": \"event\", \"value\": [\"true\"] }],\n \"limit\": 50\n}\n```\n\n## Traces for a specific model\n\n```json\n{\n \"kind\": \"TracesQuery\",\n \"dateRange\": { \"date_from\": \"-7d\" },\n \"filterTestAccounts\": true,\n \"properties\": [{ \"key\": \"$ai_model\", \"operator\": \"exact\", \"type\": \"event\", \"value\": [\"gpt-4o\"] }]\n}\n```\n\n## Traces for a specific person\n\n```json\n{\n \"kind\": \"TracesQuery\",\n \"dateRange\": { \"date_from\": \"-30d\" },\n \"personId\": \"01234567-89ab-cdef-0123-456789abcdef\",\n \"filterTestAccounts\": true\n}\n```\n\n## Random sample of traces (avoids recency bias)\n\n```json\n{\n \"kind\": \"TracesQuery\",\n \"dateRange\": { \"date_from\": \"-30d\" },\n \"filterTestAccounts\": true,\n \"randomOrder\": true,\n \"limit\": 20\n}\n```\n\n# Reminders\n\n- Ensure that any properties included are directly relevant to the context and objectives of the user's question. Avoid unnecessary or unrelated details.\n- Avoid overcomplicating the response with excessive property filters. Focus on the simplest solution.\n- This tool returns raw trace data — it does not aggregate or visualize. For aggregated LLM metrics over time (e.g. total token usage per day), use `query-trends` with AI events like `$ai_generation` instead.\n- Use `filterTestAccounts: true` by default to exclude internal users unless the user asks otherwise.\n- The default time range is last 7 days. LLM trace data tends to be recent, so shorter ranges are usually appropriate.\n- For deep inspection of a single trace (full event tree with all nested children and complete properties), use `query-llm-trace` with the trace's `id`.",
+ "description": "List LLM traces to inspect AI/LLM usage across your application. Returns traces with their events, latency, token usage, costs, errors, and other metadata. Use this tool for AI observability — debugging slow generations, investigating errors, analyzing token spend, and auditing LLM behavior.\n\nSet `detail: \"summary\"` for event metadata without any prompts or outputs when picking candidate traces, then read the one you pick with `query-llm-trace`. Omitting `detail` returns the retained properties in full, subject to size limits.\n\nUse 'read-data-schema' to discover available event properties for filtering (e.g. `$ai_model`, `$ai_provider`).\n\nExamples of use cases include:\n\n- How much are we spending on LLM tokens per day?\n- Which LLM generations are the slowest?\n- Are there any traces with errors in the last 24 hours?\n- What models are being used and how do their costs compare?\n- Show me traces for a specific user to debug their experience.\n- Are there any traces with unusually high token usage?\n\nCRITICAL: Be minimalist. Only include filters and settings that are essential to answer the user's specific question. Default settings are usually sufficient unless the user explicitly requests customization.\n\n# Data narrowing\n\n## Property filters\n\nUse property filters to narrow results. Only include property filters when they are essential to directly answer the user's question. Avoid adding them if the question can be addressed without additional segmentation and always use the minimum set of property filters needed.\n\nIMPORTANT: Do not check if a property is set unless the user explicitly asks for it.\n\nWhen using a property filter, you should:\n\n- **Prioritize properties directly related to the context or objective of the user's query.** Common AI properties include `$ai_model`, `$ai_provider`, `$ai_trace_id`, `$ai_session_id`, `$ai_latency`, `$ai_input_tokens`, `$ai_output_tokens`, `$ai_total_cost_usd`, `$ai_is_error`, `$ai_http_status`, `$ai_span_name`.\n- **Note:** `$ai_is_error` and `$ai_error` are valid filter properties but may not appear via `read-data-schema`. Use `$ai_is_error` with operator `exact` and value `[\"true\"]` to find error traces, or use `$ai_error` with `is set` to find traces with error messages.\n- **Ensure that you find both the property group and name.** Property groups should be one of the following: event, person, session, group.\n- After selecting a property, **validate that the property value accurately reflects the intended criteria**.\n- **Find the suitable operator for type** (e.g., `contains`, `is set`).\n- If the operator requires a value, use the `read-data-schema` tool to find the property values.\n\nInfer the property groups from the user's request. If your first guess doesn't yield any results, try to adjust the property group.\n\nSupported operators for the String type are:\n\n- equals (exact)\n- doesn't equal (is_not)\n- contains (icontains)\n- doesn't contain (not_icontains)\n- matches regex (regex)\n- doesn't match regex (not_regex)\n- is set\n- is not set\n\nSupported operators for the Numeric type are:\n\n- equals (exact)\n- doesn't equal (is_not)\n- greater than (gt)\n- less than (lt)\n- is set\n- is not set\n\nSupported operators for the DateTime type are:\n\n- equals (is_date_exact)\n- doesn't equal (is_not for existence check)\n- before (is_date_before)\n- after (is_date_after)\n- is set\n- is not set\n\nSupported operators for the Boolean type are:\n\n- equals\n- doesn't equal\n- is set\n- is not set\n\nAll operators take a single value except for `equals` and `doesn't equal` which can take one or more values (as an array).\n\n## Time period\n\nYou should not filter events by time using property filters. Instead, use the `dateRange` field. If the question doesn't mention time, use last 7 days as a default time period.\n\n# Traces guidelines\n\nThis is a listing tool, not a visualization/insight tool. It returns a paginated list of LLM traces — it does NOT support series, breakdowns, math aggregations, or chart types.\n\n## Response shape\n\nEach trace in the results contains:\n\n- `id` — unique trace ID\n- `traceName` — name of the trace (if set via SDK)\n- `createdAt` — timestamp of the first event in the trace\n- `distinctId` — the person's distinct ID\n- `aiSessionId` — session ID grouping related traces (e.g., a conversation)\n- `totalLatency` — total latency in seconds\n- `inputTokens` / `outputTokens` — token counts across all generations in the trace\n- `inputCost` / `outputCost` / `totalCost` — costs in USD\n- `inputState` / `outputState` — JSON input/output state of the trace (e.g., conversation messages), from the `$ai_trace` event\n- `errorCount` — number of errors in the trace\n- `isSupportTrace` — whether the trace was from a support impersonation session\n- `tools` — list of tool names called during the trace\n- `events` — list of direct child events (generations, metrics, feedback). Each event's `properties` contains the retained event data, returned in full by default and reduced to metadata under `detail: \"summary\"`, subject to response size limits. See \"Event types and their properties\" below.\n\n## Event types and their properties\n\nEach event in `events` has an `event` field indicating its type. The key properties vary by type:\n\n- **`$ai_generation`** / **`$ai_embedding`** — an LLM or embedding API call. Properties include `$ai_input` (input prompt JSON), `$ai_output_choices` (output message JSON), `$ai_model`, `$ai_provider`, `$ai_latency`, `$ai_input_tokens`, `$ai_output_tokens`, `$ai_input_cost_usd`, `$ai_output_cost_usd`, `$ai_total_cost_usd`, `$ai_tools_called`, `$ai_is_error`, `$ai_error`.\n- **`$ai_span`** — a unit of work within a trace (e.g., a retrieval step, a tool execution). Properties include `$ai_input_state`, `$ai_output_state`, `$ai_latency`, `$ai_span_name`, `$ai_parent_id`.\n- **`$ai_trace`** — the root trace event. Properties include `$ai_input_state` (e.g., conversation messages sent), `$ai_output_state` (e.g., final response), `$ai_span_name`.\n- **`$ai_metric`** — a named evaluation metric. Properties include `$ai_metric_name`, `$ai_metric_value`.\n- **`$ai_feedback`** — user-provided feedback. Properties include `$ai_feedback_text`.\n\nAll event types share `$ai_trace_id`, `$ai_span_id`, and `$ai_parent_id` for tree structure (see below).\n\n## Tree structure (IDs and parent-child relationships)\n\nEvents in a trace form a tree. Each event carries three IDs that define its position:\n\n- `$ai_trace_id` — present on every event, identifies which trace it belongs to (same as the trace's `id`)\n- `$ai_span_id` (or `$ai_generation_id` for generations) — the event's own unique identifier\n- `$ai_parent_id` — points to the parent event's `$ai_span_id`\n\nTo reconstruct the tree:\n\n1. Events where `$ai_parent_id` equals `$ai_trace_id` are **root-level children** of the trace\n2. Other events are children of the event whose `$ai_span_id` matches their `$ai_parent_id`\n3. Group events by `$ai_parent_id` and walk from root children downward\n\nGenerations (`$ai_generation`) and embeddings (`$ai_embedding`) are always leaf nodes. Spans (`$ai_span`) can have children.\n\n**Important:** This list tool only returns **direct children** of the trace (events where `$ai_parent_id` = trace ID) plus all `$ai_metric` and `$ai_feedback` events — NOT deeply nested events. For the full event tree with all nested children, use `query-llm-trace` with the trace's `id`.\n\n## Withheld properties\n\nOnly `$ai_*` properties PostHog's taxonomy defines, plus the ones first-party code writes without describing (`$ai_generation_id`, `$ai_cache_read_cost_usd`, `$ai_cache_creation_cost_usd`, `$ai_effort`) and `$session_id`, `$lib`, and `$lib_version`, reach you. Every other event property, and every person property, is withheld whichever `detail` you ask for, and its name is listed in `_redactedKeys` beside the bag. `$ai_base_url` and `$ai_request_url` arrive as the origin and path only, without the query string a provider key often sits in. A value that is not an `http` or `https` URL is withheld instead, because there is no endpoint to keep.\n\nA withheld property is unchanged in PostHog: it still works as a filter here, and you can read its value in the PostHog UI or with `execute-sql`.\n\n## Detail level\n\n`detail` controls how much of each event you get back.\n\n- `\"full\"` (default) returns every retained property in full, subject to response size limits.\n- `\"summary\"` opts into trace and event metadata only. It carries no conversation content: prompts, outputs, span states, `$ai_error`, and `$ai_feedback_text` are left out, and their names are listed in `_summaryOmittedKeys` beside the bag. The trace and span names (`traceName`, `$ai_span_name`) do come back, and an SDK can write anything into those. Use `$ai_is_error` and `$ai_http_status` to find the failed events, then read their messages with `detail: \"full\"`. A summarized trace carries `_detail: { \"mode\": \"summary\" }`.\n\nRequest `detail: \"summary\"` when finding candidate traces from their metadata; read the one you picked with `query-llm-trace` and `detail: \"full\"` when you need its content.\n\n## Response size\n\nTo protect the agent's context window, the response is bounded before it reaches you. Long string values are truncated (with a `… [truncated N chars]` marker) and oversized arrays/objects have their tail members dropped. If the combined list is still too large, trailing traces are dropped and a final `{ \"_truncated\": { \"omittedTraces\": N, ... } }` entry reports how many. When you see these markers, narrow the query (shorter date range, filters, or a smaller `limit`) or fetch a specific trace with `query-llm-trace`. The underlying data is never altered, only this response is bounded.\n\n## Pagination\n\nUse `limit` and `offset` for pagination. The default limit is 100. The response includes a `hasMore` field indicating whether more results are available.\n\n## Filtering\n\n- `filterTestAccounts` — exclude internal/test users\n- `filterSupportTraces` — exclude support impersonation traces\n- `personId` — filter by a specific person UUID\n- `groupKey` + `groupTypeIndex` — filter by a specific group\n- `randomOrder` — use random ordering instead of newest-first (useful for representative sampling)\n\n# Examples\n\n## Recent traces with errors\n\n```json\n{\n \"kind\": \"TracesQuery\",\n \"dateRange\": { \"date_from\": \"-7d\" },\n \"filterTestAccounts\": true,\n \"properties\": [{ \"key\": \"$ai_is_error\", \"operator\": \"exact\", \"type\": \"event\", \"value\": [\"true\"] }],\n \"limit\": 50\n}\n```\n\n## Traces for a specific model\n\n```json\n{\n \"kind\": \"TracesQuery\",\n \"dateRange\": { \"date_from\": \"-7d\" },\n \"filterTestAccounts\": true,\n \"properties\": [{ \"key\": \"$ai_model\", \"operator\": \"exact\", \"type\": \"event\", \"value\": [\"gpt-4o\"] }]\n}\n```\n\n## Traces for a specific person\n\n```json\n{\n \"kind\": \"TracesQuery\",\n \"dateRange\": { \"date_from\": \"-30d\" },\n \"personId\": \"01234567-89ab-cdef-0123-456789abcdef\",\n \"filterTestAccounts\": true\n}\n```\n\n## Random sample of traces (avoids recency bias)\n\n```json\n{\n \"kind\": \"TracesQuery\",\n \"dateRange\": { \"date_from\": \"-30d\" },\n \"filterTestAccounts\": true,\n \"randomOrder\": true,\n \"limit\": 20\n}\n```\n\n# Reminders\n\n- Ensure that any properties included are directly relevant to the context and objectives of the user's question. Avoid unnecessary or unrelated details.\n- Avoid overcomplicating the response with excessive property filters. Focus on the simplest solution.\n- This tool returns raw trace data — it does not aggregate or visualize. For aggregated LLM metrics over time (e.g. total token usage per day), use `query-trends` with AI events like `$ai_generation` instead.\n- Use `filterTestAccounts: true` by default to exclude internal users unless the user asks otherwise.\n- The default time range is last 7 days. LLM trace data tends to be recent, so shorter ranges are usually appropriate.\n- For deep inspection of a single trace (full event tree with all nested children and complete properties), use `query-llm-trace` with the trace's `id`.\n- When the traces show a failure that keeps happening, offer to score new traces for it automatically with an online evaluation. Load the `creating-online-evaluations` skill to set one up.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "List LLM traces",