Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(aio): keep ai observability skill references one level deep (#115846)

posthog[bot]f75cd3812598
diff --git a/products/ai_observability/skills/exploring-ai-failures/SKILL.md b/products/ai_observability/skills/exploring-ai-failures/SKILL.md
index 5a931149502c..78c02bc34f68 100644
--- a/products/ai_observability/skills/exploring-ai-failures/SKILL.md
+++ b/products/ai_observability/skills/exploring-ai-failures/SKILL.md
@@ -41,9 +41,9 @@ high volume, `exploring-llm-clusters`.
| `posthog:generate-app-url` | Build a region- and project-qualified deep link to a trace or list |
Detailed queries for each strategy below are in
-[references/finding-traces.md](references/finding-traces.md). The full `$ai_*` event schema (and the
-`events` vs `ai_events` split for heavy content like `$ai_input`/`$ai_output_choices`) lives in
-`exploring-llm-traces/references/events-and-properties.md`.
+[references/finding-traces.md](references/finding-traces.md). That file also explains the `events` vs
+`ai_events` split: heavy content like `$ai_input` / `$ai_output_choices` is not on `events`. For the
+full `$ai_*` event schema, load the `exploring-llm-traces` skill.
## Work with the user
diff --git a/products/ai_observability/skills/exploring-ai-failures/references/finding-traces.md b/products/ai_observability/skills/exploring-ai-failures/references/finding-traces.md
index a6b3e13670ea..32cce9f32cac 100644
--- a/products/ai_observability/skills/exploring-ai-failures/references/finding-traces.md
+++ b/products/ai_observability/skills/exploring-ai-failures/references/finding-traces.md
@@ -1,9 +1,41 @@
# Finding failing traces — queries
Concrete queries for each strategy in Step 2. Property names (`$ai_is_error`, `$ai_input_tokens`, …) are
-the standard AI event properties; confirm the exact ones for this project with `read-data-schema`, and
-see `exploring-llm-traces/references/events-and-properties.md` for the full schema and the `events` vs
-`ai_events` split (heavy content like `$ai_input` / `$ai_output_choices` lives on `ai_events`).
+the standard AI event properties; confirm the exact ones for this project with `read-data-schema`.
+
+## Where heavy content lives: `events` vs `ai_events`
+
+The `events` table keeps only lightweight metadata: token counts, costs, model, provider,
+`$ai_trace_id`, latency, and error flags. All queries below use it. The heavy content is **not on
+`events`**. It lives as native columns on a separate table, `posthog.ai_events`:
+
+| Heavy content | `events` property | `ai_events` column |
+| -------------- | -------------------- | ------------------ |
+| Input messages | `$ai_input` | `input` |
+| Output | `$ai_output` | `output` |
+| Output choices | `$ai_output_choices` | `output_choices` |
+| Input state | `$ai_input_state` | `input_state` |
+| Output state | `$ai_output_state` | `output_state` |
+| Tools | `$ai_tools` | `tools` |
+
+`query-llm-trace` and `query-llm-traces-list` read `posthog.ai_events` for you, so use them to read
+traces. Use SQL on `posthog.ai_events` only for batch analysis. `trace_id` is its access path, not
+`timestamp`, so filter `events` first to get trace IDs, then fetch the content by `trace_id`:
+
+```sql
+WITH matching_traces AS (
+ SELECT DISTINCT properties.$ai_trace_id AS trace_id
+ FROM events
+ WHERE event = '$ai_generation' AND timestamp >= now() - INTERVAL 7 DAY
+)
+SELECT a.trace_id, a.span_id, a.model, a.input, a.output_choices
+FROM posthog.ai_events AS a
+WHERE a.trace_id IN (SELECT trace_id FROM matching_traces)
+ORDER BY a.trace_id, a.timestamp
+```
+
+`posthog.ai_events` drops rows after the retention period (30 days by default), so older traces have
+no content.
## Discover the trace taxonomy
diff --git a/products/ai_observability/skills/exploring-llm-clusters/SKILL.md b/products/ai_observability/skills/exploring-llm-clusters/SKILL.md
index ef1597f5f954..6d32917a394a 100644
--- a/products/ai_observability/skills/exploring-llm-clusters/SKILL.md
+++ b/products/ai_observability/skills/exploring-llm-clusters/SKILL.md
@@ -232,7 +232,7 @@ ORDER BY trace_id, timestamp
```
`posthog.ai_events` has a shorter retention window than `events`; older clusters may still have metadata and metrics but no message content.
-For more detail, use the exploring LLM traces skill's [event reference](../exploring-llm-traces/references/events-and-properties.md).
+For the full `$ai_*` event schema, load the `exploring-llm-traces` skill.
## Investigation patterns
diff --git a/products/ai_observability/skills/exploring-llm-costs/SKILL.md b/products/ai_observability/skills/exploring-llm-costs/SKILL.md
index ae7091de70cd..53f2163ee837 100644
--- a/products/ai_observability/skills/exploring-llm-costs/SKILL.md
+++ b/products/ai_observability/skills/exploring-llm-costs/SKILL.md
@@ -163,7 +163,7 @@ versions for the same provider. To avoid rot:
## Tips
- Always set a time range — cost queries without one scan the full events table
-- Token, cost, model, and `$ai_trace_id` properties are on `events` — but message _content_ (`$ai_input` / `$ai_output_choices`) lives only on the `posthog.ai_events` table; see the traces skill's [event reference](../exploring-llm-traces/references/events-and-properties.md) if you need content alongside cost
+- Token, cost, model, and `$ai_trace_id` properties are on `events` — but message _content_ (`$ai_input` / `$ai_output_choices`) lives only on the `posthog.ai_events` table; load the `exploring-llm-traces` skill if you need content alongside cost
- Always include `$ai_embedding` alongside `$ai_generation` when summing cost; embeddings are cheap per-call but add up at scale
- Costs are written at ingestion (see [Calculating LLM costs](https://posthog.com/docs/ai-observability/calculating-costs)) — if `$ai_total_cost_usd` is missing or zero, read `$ai_cost_model_source` first: `passthrough` means the SDK supplied costs; `custom` means custom token prices; `openrouter` / `manual` mean automatic lookup; missing means the model wasn't matched (unusual custom model, fine-tune). Grep: `countIf(properties.$ai_total_cost_usd IS NULL)` per `(model, source)`
- Custom pricing uses **per-token** prices, not per-million — if a custom-priced model looks ~1M× too expensive or too cheap, that's almost always the bug