:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(aio): keep ai observability skill references one level deep (#115846)
posthog[bot]f75cd3812598
diff --git a/products/ai_observability/skills/exploring-ai-failures/SKILL.md b/products/ai_observability/skills/exploring-ai-failures/SKILL.mdindex 5a931149502c..78c02bc34f68 100644--- a/products/ai_observability/skills/exploring-ai-failures/SKILL.md+++ b/products/ai_observability/skills/exploring-ai-failures/SKILL.md@@ -41,9 +41,9 @@ high volume, `exploring-llm-clusters`.| `posthog:generate-app-url` | Build a region- and project-qualified deep link to a trace or list |Detailed queries for each strategy below are in-[references/finding-traces.md](references/finding-traces.md). The full `$ai_*` event schema (and the-`events` vs `ai_events` split for heavy content like `$ai_input`/`$ai_output_choices`) lives in-`exploring-llm-traces/references/events-and-properties.md`.+[references/finding-traces.md](references/finding-traces.md). That file also explains the `events` vs+`ai_events` split: heavy content like `$ai_input` / `$ai_output_choices` is not on `events`. For the+full `$ai_*` event schema, load the `exploring-llm-traces` skill.## Work with the userdiff --git a/products/ai_observability/skills/exploring-ai-failures/references/finding-traces.md b/products/ai_observability/skills/exploring-ai-failures/references/finding-traces.mdindex a6b3e13670ea..32cce9f32cac 100644--- a/products/ai_observability/skills/exploring-ai-failures/references/finding-traces.md+++ b/products/ai_observability/skills/exploring-ai-failures/references/finding-traces.md@@ -1,9 +1,41 @@# Finding failing traces — queriesConcrete queries for each strategy in Step 2. Property names (`$ai_is_error`, `$ai_input_tokens`, …) are-the standard AI event properties; confirm the exact ones for this project with `read-data-schema`, and-see `exploring-llm-traces/references/events-and-properties.md` for the full schema and the `events` vs-`ai_events` split (heavy content like `$ai_input` / `$ai_output_choices` lives on `ai_events`).+the standard AI event properties; confirm the exact ones for this project with `read-data-schema`.++## Where heavy content lives: `events` vs `ai_events`++The `events` table keeps only lightweight metadata: token counts, costs, model, provider,+`$ai_trace_id`, latency, and error flags. All queries below use it. The heavy content is **not on+`events`**. It lives as native columns on a separate table, `posthog.ai_events`:++| Heavy content | `events` property | `ai_events` column |+| -------------- | -------------------- | ------------------ |+| Input messages | `$ai_input` | `input` |+| Output | `$ai_output` | `output` |+| Output choices | `$ai_output_choices` | `output_choices` |+| Input state | `$ai_input_state` | `input_state` |+| Output state | `$ai_output_state` | `output_state` |+| Tools | `$ai_tools` | `tools` |++`query-llm-trace` and `query-llm-traces-list` read `posthog.ai_events` for you, so use them to read+traces. Use SQL on `posthog.ai_events` only for batch analysis. `trace_id` is its access path, not+`timestamp`, so filter `events` first to get trace IDs, then fetch the content by `trace_id`:++```sql+WITH matching_traces AS (+ SELECT DISTINCT properties.$ai_trace_id AS trace_id+ FROM events+ WHERE event = '$ai_generation' AND timestamp >= now() - INTERVAL 7 DAY+)+SELECT a.trace_id, a.span_id, a.model, a.input, a.output_choices+FROM posthog.ai_events AS a+WHERE a.trace_id IN (SELECT trace_id FROM matching_traces)+ORDER BY a.trace_id, a.timestamp+```++`posthog.ai_events` drops rows after the retention period (30 days by default), so older traces have+no content.## Discover the trace taxonomydiff --git a/products/ai_observability/skills/exploring-llm-clusters/SKILL.md b/products/ai_observability/skills/exploring-llm-clusters/SKILL.mdindex ef1597f5f954..6d32917a394a 100644--- a/products/ai_observability/skills/exploring-llm-clusters/SKILL.md+++ b/products/ai_observability/skills/exploring-llm-clusters/SKILL.md@@ -232,7 +232,7 @@ ORDER BY trace_id, timestamp````posthog.ai_events` has a shorter retention window than `events`; older clusters may still have metadata and metrics but no message content.-For more detail, use the exploring LLM traces skill's [event reference](../exploring-llm-traces/references/events-and-properties.md).+For the full `$ai_*` event schema, load the `exploring-llm-traces` skill.## Investigation patternsdiff --git a/products/ai_observability/skills/exploring-llm-costs/SKILL.md b/products/ai_observability/skills/exploring-llm-costs/SKILL.mdindex ae7091de70cd..53f2163ee837 100644--- a/products/ai_observability/skills/exploring-llm-costs/SKILL.md+++ b/products/ai_observability/skills/exploring-llm-costs/SKILL.md@@ -163,7 +163,7 @@ versions for the same provider. To avoid rot:## Tips- Always set a time range — cost queries without one scan the full events table-- Token, cost, model, and `$ai_trace_id` properties are on `events` — but message _content_ (`$ai_input` / `$ai_output_choices`) lives only on the `posthog.ai_events` table; see the traces skill's [event reference](../exploring-llm-traces/references/events-and-properties.md) if you need content alongside cost+- Token, cost, model, and `$ai_trace_id` properties are on `events` — but message _content_ (`$ai_input` / `$ai_output_choices`) lives only on the `posthog.ai_events` table; load the `exploring-llm-traces` skill if you need content alongside cost- Always include `$ai_embedding` alongside `$ai_generation` when summing cost; embeddings are cheap per-call but add up at scale- Costs are written at ingestion (see [Calculating LLM costs](https://posthog.com/docs/ai-observability/calculating-costs)) — if `$ai_total_cost_usd` is missing or zero, read `$ai_cost_model_source` first: `passthrough` means the SDK supplied costs; `custom` means custom token prices; `openrouter` / `manual` mean automatic lookup; missing means the model wasn't matched (unusual custom model, fine-tune). Grep: `countIf(properties.$ai_total_cost_usd IS NULL)` per `(model, source)`- Custom pricing uses **per-token** prices, not per-million — if a custom-priced model looks ~1M× too expensive or too cheap, that's almost always the bug