:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(aio): resolve cache reporting style per event in cost skill recipes (#127232)
Bernat Torresd74b5d52124e
diff --git a/products/ai_observability/skills/exploring-llm-costs/SKILL.md b/products/ai_observability/skills/exploring-llm-costs/SKILL.mdindex 53f2163ee837..65cf7045892b 100644--- a/products/ai_observability/skills/exploring-llm-costs/SKILL.md+++ b/products/ai_observability/skills/exploring-llm-costs/SKILL.md@@ -148,8 +148,9 @@ versions for the same provider. To avoid rot:- Branch on event-level flags (`$ai_cache_reporting_exclusive`,`$ai_cost_model_source`) rather than hardcoded provider or model names.- Those flags are ingestion's resolved answer for the specific event and- are the right source of truth.+ When set, those flags are ingestion's resolved answer for the specific+ event. `$ai_cache_reporting_exclusive` is often missing; see+ [cache accounting](./references/cache-accounting.md) for the fallback.- `$ai_total_cost_usd` is always authoritative for rollups — prefer itover summing components, which can drift as new cost categories areadded.@@ -170,7 +171,7 @@ versions for the same provider. To avoid rot:- Exclude errored calls from cost totals only when explicitly asked — providers still charge for many error modes, and including them gives the truthful bill- For per-user totals, exclude rows where `distinct_id = properties.$ai_trace_id` — some SDKs default distinct_id to the trace ID when no user is set- Cost is additive across `$ai_generation` + `$ai_embedding` events within a trace; summing on `$ai_span` gives zero. `$ai_trace` may carry `$ai_total_cost_usd` from some SDK wrappers — don't include it in rollups or you'll double-count. `$ai_evaluation` events also carry cost but are not part of the stock UI rollups; include them only when the user explicitly wants evaluation spend in the total-- Cache-hit rate depends on `$ai_cache_reporting_exclusive` — branch on the event-level flag rather than on provider or model name. Provider behavior and SDK versions drift; the flag is ingestion's resolved answer for that specific event+- Cache-hit rate depends on `$ai_cache_reporting_exclusive` — resolve it per event, never per group. Most events with a precomputed cost carry no flag, so never default a missing flag to inclusive; use the fallback in [cache accounting](./references/cache-accounting.md)- When answering "why is X expensive?", show the cost **and** the token split — the user almost always wants to know whether to shrink prompts, shrink outputs, or switch models- Before building a custom dashboard, check whether the stock `/ai-observability/dashboard` tiles already answer the question — re-creating them is churn- For large tenants, materialize common cost queries as insights and reuse via `insight-query`; ad-hoc SQL is fine for one-offs but re-running it on every dashboard load is expensivediff --git a/products/ai_observability/skills/exploring-llm-costs/references/breakdown-patterns.md b/products/ai_observability/skills/exploring-llm-costs/references/breakdown-patterns.mdindex 3c1e1474784e..f58855b269c4 100644--- a/products/ai_observability/skills/exploring-llm-costs/references/breakdown-patterns.md+++ b/products/ai_observability/skills/exploring-llm-costs/references/breakdown-patterns.md@@ -151,41 +151,49 @@ Split the spend to find optimization targets:```sqlposthog:execute-sqlSELECT- properties.$ai_model AS model,- round(sum(toFloat(properties.$ai_input_cost_usd)), 4) AS input_cost,- round(sum(toFloat(properties.$ai_output_cost_usd)), 4) AS output_cost,- round(sum(toFloat(properties.$ai_request_cost_usd)), 4) AS request_cost,- round(sum(toFloat(properties.$ai_web_search_cost_usd)), 4) AS web_search_cost,- round(sum(toFloat(properties.$ai_total_cost_usd)), 4) AS total_cost,- sum(toInt(properties.$ai_input_tokens)) AS input_tokens,- sum(toInt(properties.$ai_output_tokens)) AS output_tokens,- sum(toInt(properties.$ai_cache_read_input_tokens)) AS cache_read_tokens,- sum(toInt(properties.$ai_cache_creation_input_tokens)) AS cache_write_tokens,- round(- if(- any(properties.$ai_cache_reporting_exclusive) = 'true',- sum(toInt(properties.$ai_cache_read_input_tokens))- / nullIf(sum(toInt(properties.$ai_input_tokens))- + sum(toInt(properties.$ai_cache_read_input_tokens))- + sum(toInt(properties.$ai_cache_creation_input_tokens)), 0),- sum(toInt(properties.$ai_cache_read_input_tokens))- / nullIf(sum(toInt(properties.$ai_input_tokens)), 0)- ), 3- ) AS cache_hit_rate-FROM events-WHERE event = '$ai_generation'- AND timestamp >= now() - INTERVAL 30 DAY+ model,+ round(sum(input_cost), 4) AS input_cost,+ round(sum(output_cost), 4) AS output_cost,+ round(sum(request_cost), 4) AS request_cost,+ round(sum(web_search_cost), 4) AS web_search_cost,+ round(sum(total_cost), 4) AS total_cost,+ sum(input) AS input_tokens,+ sum(output) AS output_tokens,+ sum(cache_read) AS cache_read_tokens,+ sum(cache_write) AS cache_write_tokens,+ round(sum(cache_read) / nullIf(sum(if(exclusive, input + cache_read + cache_write, input)), 0), 3) AS cache_hit_rate+FROM (+ SELECT+ properties.$ai_model AS model,+ toFloat(properties.$ai_input_cost_usd) AS input_cost,+ toFloat(properties.$ai_output_cost_usd) AS output_cost,+ toFloat(properties.$ai_request_cost_usd) AS request_cost,+ toFloat(properties.$ai_web_search_cost_usd) AS web_search_cost,+ toFloat(properties.$ai_total_cost_usd) AS total_cost,+ ifNull(toInt(properties.$ai_input_tokens), 0) AS input,+ ifNull(toInt(properties.$ai_output_tokens), 0) AS output,+ ifNull(toInt(properties.$ai_cache_read_input_tokens), 0) AS cache_read,+ ifNull(toInt(properties.$ai_cache_creation_input_tokens), 0) AS cache_write,+ lower(ifNull(properties.$ai_provider, '')) AS provider,+ properties.$ai_cache_reporting_exclusive IS NULL AS flag_missing,+ (provider = 'anthropic'+ OR positionCaseInsensitive(ifNull(properties.$ai_model, ''), 'anthropic') > 0+ OR startsWith(lower(ifNull(properties.$ai_model, '')), 'claude'))+ AND NOT (provider = 'gateway' AND lower(ifNull(properties.$ai_framework, '')) = 'vercel') AS anthropic_style,+ if(flag_missing, anthropic_style, toString(properties.$ai_cache_reporting_exclusive) = 'true')+ OR input < cache_read + cache_write AS exclusive+ FROM events+ WHERE event = '$ai_generation'+ AND timestamp >= now() - INTERVAL 30 DAY+)GROUP BY modelORDER BY total_cost DESC```-The `cache_hit_rate` uses the provider-aware formula from-[cache accounting](./cache-accounting.md) — it branches on-`$ai_cache_reporting_exclusive` so the denominator is correct for both-exclusive and inclusive providers without hardcoding any provider or model-names. If a single model mixes both reporting styles across events-(unusual), split by `$ai_cache_reporting_exclusive` in the GROUP BY-instead of `any()`.+The `cache_hit_rate` resolves the reporting style per event, as described+in [cache accounting](./cache-accounting.md), so the denominator is correct+for both exclusive and inclusive providers, including events that carry no+`$ai_cache_reporting_exclusive` flag.Rank and roll up on `total_cost` — summing only the input/output componentsdrops request and web-search fees and can diverge from the `/ai-observability`diff --git a/products/ai_observability/skills/exploring-llm-costs/references/cache-accounting.md b/products/ai_observability/skills/exploring-llm-costs/references/cache-accounting.mdindex 6c9f49220400..e44db7afa945 100644--- a/products/ai_observability/skills/exploring-llm-costs/references/cache-accounting.md+++ b/products/ai_observability/skills/exploring-llm-costs/references/cache-accounting.md@@ -11,31 +11,67 @@ changes accordingly:Don't hardcode provider behavior — it varies by SDK and by SDK version,and providers can change their own reporting style over time. Instead,-trust the per-event flag: ingestion auto-detects and writes the resolved-value to `$ai_cache_reporting_exclusive` (boolean) on every-`$ai_generation`. Callers can also override with-`$ai_cache_reporting_exclusive: true|false` when manually capturing.+trust the per-event flag `$ai_cache_reporting_exclusive` (boolean).+Ingestion writes it when it calculates the event's cost, and callers can+set it explicitly when manually capturing.-## Cache-hit rate, branching on the per-event flag+The flag is often missing. Ingestion writes it only when it calculates+the cost, so events with a precomputed (`passthrough`) cost, an unknown+model, or no input token count carry no flag. For those, apply the same+fallback ingestion uses: Anthropic-style events (provider `anthropic`, or a+model name with `anthropic` or starting with `claude`, except through the+Vercel AI Gateway) are exclusive.++Every event is also exclusive when `input_tokens < cache_read ++cache_creation`, since an inclusive count can never be smaller than its+cache tokens. This check also corrects a flag set to `false` by mistake,+and it keeps the rate at or below 1.++## Cache-hit rate, resolved per event++Resolve the reporting style on each event, then aggregate. Picking one+style per group (`any(flag)`, or treating a missing flag as inclusive)+applies the wrong denominator to part of the group and can push the rate+above 1.```sqlposthog:execute-sqlSELECT- properties.$ai_model AS model,- if(properties.$ai_cache_reporting_exclusive = 'true',- sum(toInt(properties.$ai_cache_read_input_tokens))- / nullIf(sum(toInt(properties.$ai_input_tokens))- + sum(toInt(properties.$ai_cache_read_input_tokens))- + sum(toInt(properties.$ai_cache_creation_input_tokens)), 0),- sum(toInt(properties.$ai_cache_read_input_tokens))- / nullIf(sum(toInt(properties.$ai_input_tokens)), 0)- ) AS cache_hit_rate-FROM events-WHERE event = '$ai_generation'- AND timestamp >= now() - INTERVAL 30 DAY-GROUP BY model, properties.$ai_cache_reporting_exclusive+ model,+ round(sum(cache_read) / nullIf(sum(if(exclusive, input + cache_read + cache_write, input)), 0), 3) AS cache_hit_rate,+ countIf(flag_missing) AS calls_with_inferred_flag,+ count() AS calls+FROM (+ SELECT+ properties.$ai_model AS model,+ ifNull(toInt(properties.$ai_input_tokens), 0) AS input,+ ifNull(toInt(properties.$ai_cache_read_input_tokens), 0) AS cache_read,+ ifNull(toInt(properties.$ai_cache_creation_input_tokens), 0) AS cache_write,+ lower(ifNull(properties.$ai_provider, '')) AS provider,+ properties.$ai_cache_reporting_exclusive IS NULL AS flag_missing,+ (provider = 'anthropic'+ OR positionCaseInsensitive(ifNull(properties.$ai_model, ''), 'anthropic') > 0+ OR startsWith(lower(ifNull(properties.$ai_model, '')), 'claude'))+ AND NOT (provider = 'gateway' AND lower(ifNull(properties.$ai_framework, '')) = 'vercel') AS anthropic_style,+ if(flag_missing, anthropic_style, toString(properties.$ai_cache_reporting_exclusive) = 'true')+ OR input < cache_read + cache_write AS exclusive+ FROM events+ WHERE event = '$ai_generation'+ AND timestamp >= now() - INTERVAL 30 DAY+)+GROUP BY model```-The same provider-aware `if(...)` formula is what powers `cache_hit_rate`-in the [breakdown patterns](./breakdown-patterns.md) "input vs output vs-cache economics" recipe.+Compare the flag through `toString(...) = 'true'` behind an `IS NULL`+check. Its type depends on the project's property definition (boolean or+string), and `toString` of a missing value returns `''`, not null.++The inferred fallback can still miss: an exclusive event from another+provider, whose fresh input outweighs its cache tokens, reads as+inclusive, which overstates its rate. When `calls_with_inferred_flag` is a+large share of `calls`, say the rate is partly inferred.++The [breakdown patterns](./breakdown-patterns.md) "input vs output vs+cache economics" recipe and the+[regression debugging](./regression-debugging.md) cache step use this+same per-event resolution.diff --git a/products/ai_observability/skills/exploring-llm-costs/references/regression-debugging.md b/products/ai_observability/skills/exploring-llm-costs/references/regression-debugging.mdindex 854390c2b2ff..22319356ba92 100644--- a/products/ai_observability/skills/exploring-llm-costs/references/regression-debugging.md+++ b/products/ai_observability/skills/exploring-llm-costs/references/regression-debugging.md@@ -64,31 +64,40 @@ ORDER BY day, model```sqlposthog:execute-sqlSELECT- toDate(timestamp) AS day,- properties.$ai_model AS model,- round(- if(- any(properties.$ai_cache_reporting_exclusive) = 'true',- sum(toInt(properties.$ai_cache_read_input_tokens))- / nullIf(sum(toInt(properties.$ai_input_tokens))- + sum(toInt(properties.$ai_cache_read_input_tokens))- + sum(toInt(properties.$ai_cache_creation_input_tokens)), 0),- sum(toInt(properties.$ai_cache_read_input_tokens))- / nullIf(sum(toInt(properties.$ai_input_tokens)), 0)- ), 3- ) AS cache_hit_rate,- sum(toInt(properties.$ai_input_tokens)) AS input_tokens,- sum(toInt(properties.$ai_cache_read_input_tokens)) AS cache_read_tokens-FROM events-WHERE event = '$ai_generation'- AND timestamp >= now() - INTERVAL 30 DAY+ day,+ model,+ round(sum(cache_read) / nullIf(sum(if(exclusive, input + cache_read + cache_write, input)), 0), 3) AS cache_hit_rate,+ sum(input) AS input_tokens,+ sum(cache_read) AS cache_read_tokens,+ countIf(flag_missing) AS calls_with_inferred_flag,+ count() AS calls+FROM (+ SELECT+ toDate(timestamp) AS day,+ properties.$ai_model AS model,+ ifNull(toInt(properties.$ai_input_tokens), 0) AS input,+ ifNull(toInt(properties.$ai_cache_read_input_tokens), 0) AS cache_read,+ ifNull(toInt(properties.$ai_cache_creation_input_tokens), 0) AS cache_write,+ lower(ifNull(properties.$ai_provider, '')) AS provider,+ properties.$ai_cache_reporting_exclusive IS NULL AS flag_missing,+ (provider = 'anthropic'+ OR positionCaseInsensitive(ifNull(properties.$ai_model, ''), 'anthropic') > 0+ OR startsWith(lower(ifNull(properties.$ai_model, '')), 'claude'))+ AND NOT (provider = 'gateway' AND lower(ifNull(properties.$ai_framework, '')) = 'vercel') AS anthropic_style,+ if(flag_missing, anthropic_style, toString(properties.$ai_cache_reporting_exclusive) = 'true')+ OR input < cache_read + cache_write AS exclusive+ FROM events+ WHERE event = '$ai_generation'+ AND timestamp >= now() - INTERVAL 30 DAY+)GROUP BY day, modelORDER BY day, model```A drop often follows a system-prompt change that invalidated the cached prefix.-A rate above 1 means the flag is unset on those events, so the inclusive-branch ran over exclusive data. Read the token columns instead.+The rate resolves the reporting style per event; see+[cache accounting](./cache-accounting.md) for why, and for how to read+`calls_with_inferred_flag`.## Step 5 — Isolate the feature