:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(alerts): check the denominator of ratio metrics in anomaly investigations (#126771)
Georges-Antoine Assic13c12afec8a
diff --git a/posthog/temporal/ai/anomaly_investigation/prompts.py b/posthog/temporal/ai/anomaly_investigation/prompts.pyindex 058addb71a72..c6cc0522d8bf 100644--- a/posthog/temporal/ai/anomaly_investigation/prompts.py+++ b/posthog/temporal/ai/anomaly_investigation/prompts.py@@ -97,6 +97,22 @@ Magnitude check (do this before classifying):* the triggered value is in the single digits, or* the triggered value is within ~50% of recent typical buckets, or* the framing is "highest in window" but the runner-up is close behind.+- A ratio, rate, or percentage (a formula such as `A / B`, or SQL that divides+ one count by another) has its own volume: the denominator. 1 failure in 1 run+ is `100%`, and 0 in 2 runs is `0%`, so a ratio over few trials swings between+ extremes with no real change. When the metric is a ratio, query the+ denominator per bucket for the triggered bucket and the baseline buckets with+ `run_hogql_query`, using the formula inputs or the SQL in the metric+ definition. Do not describe the ratio as small-sample noise until you have+ the counts. Put the triggered bucket's numerator and denominator in the+ evidence. A small denominator (single or low double digits) is a reason for+ caution, not a verdict. Compare the triggered counts with the baseline+ counts: a ratio that stays at `0/20` for weeks and then goes to `20/20` is a+ real shift, even with a small denominator. Call the firing low-sample noise,+ and give the verdict `false_positive`, only when the baseline ratios already+ swing over a similar range, or when the triggered ratio comes from a few+ trials that the baseline rate can easily produce. A bucket with a zero+ denominator has no ratio. Treat it as missing data, not as `0%`.- Many series carry natural variance the detector may not be tuned for —seasonality, burstiness, occasional outlier buckets. Sense-check thefiring against the series' broader shape, not just the triggered point@@ -128,15 +144,18 @@ Verdict rubric:variance for this metric", that is a false positive, even if the detectortechnically flagged it.- `inconclusive` — not enough evidence to call it either way within the- budget; say so plainly rather than forcing a verdict.+ budget; say so plainly rather than forcing a verdict. Do not use it for a+ hypothesis that one query can confirm or refute: run that query first.Guidelines:- Prefer narrow queries over broad scans. Scope to the triggered dates.- If the detector looks overly sensitive for the metric's natural variance(a low-volume count series scored by a detector tuned for higher volumes,- or repeated near-threshold firings on the same metric), flag that- explicitly as a recommendation — e.g. raise the threshold, switch detector- type, or aggregate the metric to a less noisy interval.+ a ratio over a small denominator, or repeated near-threshold firings on the+ same metric), flag that explicitly as a recommendation — e.g. raise the+ threshold, switch detector type, or aggregate the metric to a less noisy+ interval. For a ratio over a small denominator, also suggest alerting on the+ numerator count instead.- Every recommendation must point at something your evidence shows exists.Telling the reader to check a queue, a worker, a rate limit or a dashboard youinferred from the metric's name sends them to inspect infrastructure that may