Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(alerts): check the denominator of ratio metrics in anomaly investigations (#126771)

Georges-Antoine Assic13c12afec8a
diff --git a/posthog/temporal/ai/anomaly_investigation/prompts.py b/posthog/temporal/ai/anomaly_investigation/prompts.py
index 058addb71a72..c6cc0522d8bf 100644
--- a/posthog/temporal/ai/anomaly_investigation/prompts.py
+++ b/posthog/temporal/ai/anomaly_investigation/prompts.py
@@ -97,6 +97,22 @@ Magnitude check (do this before classifying):
* the triggered value is in the single digits, or
* the triggered value is within ~50% of recent typical buckets, or
* the framing is "highest in window" but the runner-up is close behind.
+- A ratio, rate, or percentage (a formula such as `A / B`, or SQL that divides
+ one count by another) has its own volume: the denominator. 1 failure in 1 run
+ is `100%`, and 0 in 2 runs is `0%`, so a ratio over few trials swings between
+ extremes with no real change. When the metric is a ratio, query the
+ denominator per bucket for the triggered bucket and the baseline buckets with
+ `run_hogql_query`, using the formula inputs or the SQL in the metric
+ definition. Do not describe the ratio as small-sample noise until you have
+ the counts. Put the triggered bucket's numerator and denominator in the
+ evidence. A small denominator (single or low double digits) is a reason for
+ caution, not a verdict. Compare the triggered counts with the baseline
+ counts: a ratio that stays at `0/20` for weeks and then goes to `20/20` is a
+ real shift, even with a small denominator. Call the firing low-sample noise,
+ and give the verdict `false_positive`, only when the baseline ratios already
+ swing over a similar range, or when the triggered ratio comes from a few
+ trials that the baseline rate can easily produce. A bucket with a zero
+ denominator has no ratio. Treat it as missing data, not as `0%`.
- Many series carry natural variance the detector may not be tuned for —
seasonality, burstiness, occasional outlier buckets. Sense-check the
firing against the series' broader shape, not just the triggered point
@@ -128,15 +144,18 @@ Verdict rubric:
variance for this metric", that is a false positive, even if the detector
technically flagged it.
- `inconclusive` — not enough evidence to call it either way within the
- budget; say so plainly rather than forcing a verdict.
+ budget; say so plainly rather than forcing a verdict. Do not use it for a
+ hypothesis that one query can confirm or refute: run that query first.
Guidelines:
- Prefer narrow queries over broad scans. Scope to the triggered dates.
- If the detector looks overly sensitive for the metric's natural variance
(a low-volume count series scored by a detector tuned for higher volumes,
- or repeated near-threshold firings on the same metric), flag that
- explicitly as a recommendation — e.g. raise the threshold, switch detector
- type, or aggregate the metric to a less noisy interval.
+ a ratio over a small denominator, or repeated near-threshold firings on the
+ same metric), flag that explicitly as a recommendation — e.g. raise the
+ threshold, switch detector type, or aggregate the metric to a less noisy
+ interval. For a ratio over a small denominator, also suggest alerting on the
+ numerator count instead.
- Every recommendation must point at something your evidence shows exists.
Telling the reader to check a queue, a worker, a rate limit or a dashboard you
inferred from the metric's name sends them to inspect infrastructure that may