Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(aio): resolve cache reporting style per event in cost skill recipes (#127232)

Bernat Torresd74b5d52124e
diff --git a/products/ai_observability/skills/exploring-llm-costs/SKILL.md b/products/ai_observability/skills/exploring-llm-costs/SKILL.md
index 53f2163ee837..65cf7045892b 100644
--- a/products/ai_observability/skills/exploring-llm-costs/SKILL.md
+++ b/products/ai_observability/skills/exploring-llm-costs/SKILL.md
@@ -148,8 +148,9 @@ versions for the same provider. To avoid rot:
- Branch on event-level flags (`$ai_cache_reporting_exclusive`,
`$ai_cost_model_source`) rather than hardcoded provider or model names.
- Those flags are ingestion's resolved answer for the specific event and
- are the right source of truth.
+ When set, those flags are ingestion's resolved answer for the specific
+ event. `$ai_cache_reporting_exclusive` is often missing; see
+ [cache accounting](./references/cache-accounting.md) for the fallback.
- `$ai_total_cost_usd` is always authoritative for rollups — prefer it
over summing components, which can drift as new cost categories are
added.
@@ -170,7 +171,7 @@ versions for the same provider. To avoid rot:
- Exclude errored calls from cost totals only when explicitly asked — providers still charge for many error modes, and including them gives the truthful bill
- For per-user totals, exclude rows where `distinct_id = properties.$ai_trace_id` — some SDKs default distinct_id to the trace ID when no user is set
- Cost is additive across `$ai_generation` + `$ai_embedding` events within a trace; summing on `$ai_span` gives zero. `$ai_trace` may carry `$ai_total_cost_usd` from some SDK wrappers — don't include it in rollups or you'll double-count. `$ai_evaluation` events also carry cost but are not part of the stock UI rollups; include them only when the user explicitly wants evaluation spend in the total
-- Cache-hit rate depends on `$ai_cache_reporting_exclusive` — branch on the event-level flag rather than on provider or model name. Provider behavior and SDK versions drift; the flag is ingestion's resolved answer for that specific event
+- Cache-hit rate depends on `$ai_cache_reporting_exclusive` — resolve it per event, never per group. Most events with a precomputed cost carry no flag, so never default a missing flag to inclusive; use the fallback in [cache accounting](./references/cache-accounting.md)
- When answering "why is X expensive?", show the cost **and** the token split — the user almost always wants to know whether to shrink prompts, shrink outputs, or switch models
- Before building a custom dashboard, check whether the stock `/ai-observability/dashboard` tiles already answer the question — re-creating them is churn
- For large tenants, materialize common cost queries as insights and reuse via `insight-query`; ad-hoc SQL is fine for one-offs but re-running it on every dashboard load is expensive
diff --git a/products/ai_observability/skills/exploring-llm-costs/references/breakdown-patterns.md b/products/ai_observability/skills/exploring-llm-costs/references/breakdown-patterns.md
index 3c1e1474784e..f58855b269c4 100644
--- a/products/ai_observability/skills/exploring-llm-costs/references/breakdown-patterns.md
+++ b/products/ai_observability/skills/exploring-llm-costs/references/breakdown-patterns.md
@@ -151,41 +151,49 @@ Split the spend to find optimization targets:
```sql
posthog:execute-sql
SELECT
- properties.$ai_model AS model,
- round(sum(toFloat(properties.$ai_input_cost_usd)), 4) AS input_cost,
- round(sum(toFloat(properties.$ai_output_cost_usd)), 4) AS output_cost,
- round(sum(toFloat(properties.$ai_request_cost_usd)), 4) AS request_cost,
- round(sum(toFloat(properties.$ai_web_search_cost_usd)), 4) AS web_search_cost,
- round(sum(toFloat(properties.$ai_total_cost_usd)), 4) AS total_cost,
- sum(toInt(properties.$ai_input_tokens)) AS input_tokens,
- sum(toInt(properties.$ai_output_tokens)) AS output_tokens,
- sum(toInt(properties.$ai_cache_read_input_tokens)) AS cache_read_tokens,
- sum(toInt(properties.$ai_cache_creation_input_tokens)) AS cache_write_tokens,
- round(
- if(
- any(properties.$ai_cache_reporting_exclusive) = 'true',
- sum(toInt(properties.$ai_cache_read_input_tokens))
- / nullIf(sum(toInt(properties.$ai_input_tokens))
- + sum(toInt(properties.$ai_cache_read_input_tokens))
- + sum(toInt(properties.$ai_cache_creation_input_tokens)), 0),
- sum(toInt(properties.$ai_cache_read_input_tokens))
- / nullIf(sum(toInt(properties.$ai_input_tokens)), 0)
- ), 3
- ) AS cache_hit_rate
-FROM events
-WHERE event = '$ai_generation'
- AND timestamp >= now() - INTERVAL 30 DAY
+ model,
+ round(sum(input_cost), 4) AS input_cost,
+ round(sum(output_cost), 4) AS output_cost,
+ round(sum(request_cost), 4) AS request_cost,
+ round(sum(web_search_cost), 4) AS web_search_cost,
+ round(sum(total_cost), 4) AS total_cost,
+ sum(input) AS input_tokens,
+ sum(output) AS output_tokens,
+ sum(cache_read) AS cache_read_tokens,
+ sum(cache_write) AS cache_write_tokens,
+ round(sum(cache_read) / nullIf(sum(if(exclusive, input + cache_read + cache_write, input)), 0), 3) AS cache_hit_rate
+FROM (
+ SELECT
+ properties.$ai_model AS model,
+ toFloat(properties.$ai_input_cost_usd) AS input_cost,
+ toFloat(properties.$ai_output_cost_usd) AS output_cost,
+ toFloat(properties.$ai_request_cost_usd) AS request_cost,
+ toFloat(properties.$ai_web_search_cost_usd) AS web_search_cost,
+ toFloat(properties.$ai_total_cost_usd) AS total_cost,
+ ifNull(toInt(properties.$ai_input_tokens), 0) AS input,
+ ifNull(toInt(properties.$ai_output_tokens), 0) AS output,
+ ifNull(toInt(properties.$ai_cache_read_input_tokens), 0) AS cache_read,
+ ifNull(toInt(properties.$ai_cache_creation_input_tokens), 0) AS cache_write,
+ lower(ifNull(properties.$ai_provider, '')) AS provider,
+ properties.$ai_cache_reporting_exclusive IS NULL AS flag_missing,
+ (provider = 'anthropic'
+ OR positionCaseInsensitive(ifNull(properties.$ai_model, ''), 'anthropic') > 0
+ OR startsWith(lower(ifNull(properties.$ai_model, '')), 'claude'))
+ AND NOT (provider = 'gateway' AND lower(ifNull(properties.$ai_framework, '')) = 'vercel') AS anthropic_style,
+ if(flag_missing, anthropic_style, toString(properties.$ai_cache_reporting_exclusive) = 'true')
+ OR input < cache_read + cache_write AS exclusive
+ FROM events
+ WHERE event = '$ai_generation'
+ AND timestamp >= now() - INTERVAL 30 DAY
+)
GROUP BY model
ORDER BY total_cost DESC
```
-The `cache_hit_rate` uses the provider-aware formula from
-[cache accounting](./cache-accounting.md) — it branches on
-`$ai_cache_reporting_exclusive` so the denominator is correct for both
-exclusive and inclusive providers without hardcoding any provider or model
-names. If a single model mixes both reporting styles across events
-(unusual), split by `$ai_cache_reporting_exclusive` in the GROUP BY
-instead of `any()`.
+The `cache_hit_rate` resolves the reporting style per event, as described
+in [cache accounting](./cache-accounting.md), so the denominator is correct
+for both exclusive and inclusive providers, including events that carry no
+`$ai_cache_reporting_exclusive` flag.
Rank and roll up on `total_cost` — summing only the input/output components
drops request and web-search fees and can diverge from the `/ai-observability`
diff --git a/products/ai_observability/skills/exploring-llm-costs/references/cache-accounting.md b/products/ai_observability/skills/exploring-llm-costs/references/cache-accounting.md
index 6c9f49220400..e44db7afa945 100644
--- a/products/ai_observability/skills/exploring-llm-costs/references/cache-accounting.md
+++ b/products/ai_observability/skills/exploring-llm-costs/references/cache-accounting.md
@@ -11,31 +11,67 @@ changes accordingly:
Don't hardcode provider behavior — it varies by SDK and by SDK version,
and providers can change their own reporting style over time. Instead,
-trust the per-event flag: ingestion auto-detects and writes the resolved
-value to `$ai_cache_reporting_exclusive` (boolean) on every
-`$ai_generation`. Callers can also override with
-`$ai_cache_reporting_exclusive: true|false` when manually capturing.
+trust the per-event flag `$ai_cache_reporting_exclusive` (boolean).
+Ingestion writes it when it calculates the event's cost, and callers can
+set it explicitly when manually capturing.
-## Cache-hit rate, branching on the per-event flag
+The flag is often missing. Ingestion writes it only when it calculates
+the cost, so events with a precomputed (`passthrough`) cost, an unknown
+model, or no input token count carry no flag. For those, apply the same
+fallback ingestion uses: Anthropic-style events (provider `anthropic`, or a
+model name with `anthropic` or starting with `claude`, except through the
+Vercel AI Gateway) are exclusive.
+
+Every event is also exclusive when `input_tokens < cache_read +
+cache_creation`, since an inclusive count can never be smaller than its
+cache tokens. This check also corrects a flag set to `false` by mistake,
+and it keeps the rate at or below 1.
+
+## Cache-hit rate, resolved per event
+
+Resolve the reporting style on each event, then aggregate. Picking one
+style per group (`any(flag)`, or treating a missing flag as inclusive)
+applies the wrong denominator to part of the group and can push the rate
+above 1.
```sql
posthog:execute-sql
SELECT
- properties.$ai_model AS model,
- if(properties.$ai_cache_reporting_exclusive = 'true',
- sum(toInt(properties.$ai_cache_read_input_tokens))
- / nullIf(sum(toInt(properties.$ai_input_tokens))
- + sum(toInt(properties.$ai_cache_read_input_tokens))
- + sum(toInt(properties.$ai_cache_creation_input_tokens)), 0),
- sum(toInt(properties.$ai_cache_read_input_tokens))
- / nullIf(sum(toInt(properties.$ai_input_tokens)), 0)
- ) AS cache_hit_rate
-FROM events
-WHERE event = '$ai_generation'
- AND timestamp >= now() - INTERVAL 30 DAY
-GROUP BY model, properties.$ai_cache_reporting_exclusive
+ model,
+ round(sum(cache_read) / nullIf(sum(if(exclusive, input + cache_read + cache_write, input)), 0), 3) AS cache_hit_rate,
+ countIf(flag_missing) AS calls_with_inferred_flag,
+ count() AS calls
+FROM (
+ SELECT
+ properties.$ai_model AS model,
+ ifNull(toInt(properties.$ai_input_tokens), 0) AS input,
+ ifNull(toInt(properties.$ai_cache_read_input_tokens), 0) AS cache_read,
+ ifNull(toInt(properties.$ai_cache_creation_input_tokens), 0) AS cache_write,
+ lower(ifNull(properties.$ai_provider, '')) AS provider,
+ properties.$ai_cache_reporting_exclusive IS NULL AS flag_missing,
+ (provider = 'anthropic'
+ OR positionCaseInsensitive(ifNull(properties.$ai_model, ''), 'anthropic') > 0
+ OR startsWith(lower(ifNull(properties.$ai_model, '')), 'claude'))
+ AND NOT (provider = 'gateway' AND lower(ifNull(properties.$ai_framework, '')) = 'vercel') AS anthropic_style,
+ if(flag_missing, anthropic_style, toString(properties.$ai_cache_reporting_exclusive) = 'true')
+ OR input < cache_read + cache_write AS exclusive
+ FROM events
+ WHERE event = '$ai_generation'
+ AND timestamp >= now() - INTERVAL 30 DAY
+)
+GROUP BY model
```
-The same provider-aware `if(...)` formula is what powers `cache_hit_rate`
-in the [breakdown patterns](./breakdown-patterns.md) "input vs output vs
-cache economics" recipe.
+Compare the flag through `toString(...) = 'true'` behind an `IS NULL`
+check. Its type depends on the project's property definition (boolean or
+string), and `toString` of a missing value returns `''`, not null.
+
+The inferred fallback can still miss: an exclusive event from another
+provider, whose fresh input outweighs its cache tokens, reads as
+inclusive, which overstates its rate. When `calls_with_inferred_flag` is a
+large share of `calls`, say the rate is partly inferred.
+
+The [breakdown patterns](./breakdown-patterns.md) "input vs output vs
+cache economics" recipe and the
+[regression debugging](./regression-debugging.md) cache step use this
+same per-event resolution.
diff --git a/products/ai_observability/skills/exploring-llm-costs/references/regression-debugging.md b/products/ai_observability/skills/exploring-llm-costs/references/regression-debugging.md
index 854390c2b2ff..22319356ba92 100644
--- a/products/ai_observability/skills/exploring-llm-costs/references/regression-debugging.md
+++ b/products/ai_observability/skills/exploring-llm-costs/references/regression-debugging.md
@@ -64,31 +64,40 @@ ORDER BY day, model
```sql
posthog:execute-sql
SELECT
- toDate(timestamp) AS day,
- properties.$ai_model AS model,
- round(
- if(
- any(properties.$ai_cache_reporting_exclusive) = 'true',
- sum(toInt(properties.$ai_cache_read_input_tokens))
- / nullIf(sum(toInt(properties.$ai_input_tokens))
- + sum(toInt(properties.$ai_cache_read_input_tokens))
- + sum(toInt(properties.$ai_cache_creation_input_tokens)), 0),
- sum(toInt(properties.$ai_cache_read_input_tokens))
- / nullIf(sum(toInt(properties.$ai_input_tokens)), 0)
- ), 3
- ) AS cache_hit_rate,
- sum(toInt(properties.$ai_input_tokens)) AS input_tokens,
- sum(toInt(properties.$ai_cache_read_input_tokens)) AS cache_read_tokens
-FROM events
-WHERE event = '$ai_generation'
- AND timestamp >= now() - INTERVAL 30 DAY
+ day,
+ model,
+ round(sum(cache_read) / nullIf(sum(if(exclusive, input + cache_read + cache_write, input)), 0), 3) AS cache_hit_rate,
+ sum(input) AS input_tokens,
+ sum(cache_read) AS cache_read_tokens,
+ countIf(flag_missing) AS calls_with_inferred_flag,
+ count() AS calls
+FROM (
+ SELECT
+ toDate(timestamp) AS day,
+ properties.$ai_model AS model,
+ ifNull(toInt(properties.$ai_input_tokens), 0) AS input,
+ ifNull(toInt(properties.$ai_cache_read_input_tokens), 0) AS cache_read,
+ ifNull(toInt(properties.$ai_cache_creation_input_tokens), 0) AS cache_write,
+ lower(ifNull(properties.$ai_provider, '')) AS provider,
+ properties.$ai_cache_reporting_exclusive IS NULL AS flag_missing,
+ (provider = 'anthropic'
+ OR positionCaseInsensitive(ifNull(properties.$ai_model, ''), 'anthropic') > 0
+ OR startsWith(lower(ifNull(properties.$ai_model, '')), 'claude'))
+ AND NOT (provider = 'gateway' AND lower(ifNull(properties.$ai_framework, '')) = 'vercel') AS anthropic_style,
+ if(flag_missing, anthropic_style, toString(properties.$ai_cache_reporting_exclusive) = 'true')
+ OR input < cache_read + cache_write AS exclusive
+ FROM events
+ WHERE event = '$ai_generation'
+ AND timestamp >= now() - INTERVAL 30 DAY
+)
GROUP BY day, model
ORDER BY day, model
```
A drop often follows a system-prompt change that invalidated the cached prefix.
-A rate above 1 means the flag is unset on those events, so the inclusive
-branch ran over exclusive data. Read the token columns instead.
+The rate resolves the reporting style per event; see
+[cache accounting](./cache-accounting.md) for why, and for how to read
+`calls_with_inferred_flag`.
## Step 5 — Isolate the feature