:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(signals): evaluate count-based monitoring rates (#126576)
Mikayla Thompsona845178f78f6
diff --git a/docs/internal/signals-pr-lifecycle.md b/docs/internal/signals-pr-lifecycle.mdindex 24cb5412c005..df4f4fc9e9b4 100644--- a/docs/internal/signals-pr-lifecycle.md+++ b/docs/internal/signals-pr-lifecycle.md@@ -123,7 +123,8 @@ Disabling Monitoring stops execution without starting a post-resolution scheduleAfter each metric check's soak ends, the coordinator evaluates its original Monitoring-start-to-deadline window, even if execution runs late.Final work uses the current attempt's check membership and definition versions, independently of interim observations and recurring run budgets.Candidate collection scans past stale definitions until it fills the eligible batch or reaches the tick deadline.-Single-source raw event/action counts and event-property sums have supported evidence adapters; sampled values, formulas, distinct counts, and session/property aggregates without an adapter produce explicit unsupported results.+Single-source raw event/action counts and event-property sums have supported evidence adapters; sampled values, arbitrary formulas, distinct counts, and session/property aggregates without an adapter produce explicit unsupported results.+Ratios of two raw event/action counts support `A / B` and `B / A`, optionally multiplied by 100, with fixed targets. The evaluator fetches both raw counts in one Trends response and calculates the formula from those exact values. Qualification uses the denominator from that same response, preserving series filters, shared population filters, team, and fixed window. The response must be fresh through the window's end. Cached formula results cannot be combined with independently refreshed denominators, including after budget deferrals. Any positive denominator qualifies, including when the numerator is zero; a zero denominator remains insufficient even if Trends returns zero for division by zero. This policy evaluates the authored target without claiming statistical confidence. Formula encodings (`formula`, `formulas`, and `formulaNodes`) follow Trends precedence. Other arithmetic and aggregates remain unsupported.Session aggregation remains unsupported when a count series carries `$session_duration` or a `session_properties` property type.An average without a source-specific minimum denominator produces insufficient evidence.Positive counts and supported sums qualify as sufficient evidence; zero totals remain insufficient when no relevant activity source is defined, even if they meet the numeric target.@@ -167,7 +168,7 @@ Enabled research can author `run_interval_minutes` for a metric check with a Mon### Evidence qualification contract-`report_check_evidence.qualify_metric_evidence` is a pure contract for future final Monitoring evaluators. It has no production callers and changes neither scheduling nor interim observations. Qualification is independent of the target comparison: sufficient evidence can demonstrate a failed expectation, while a matching value with insufficient evidence cannot establish success.+`report_check_evidence.qualify_metric_evidence` is the pure qualification contract used by final Monitoring metric evaluation. It changes neither scheduling nor interim observations. Qualification is independent of the target comparison: sufficient evidence can demonstrate a failed expectation, while a matching value with insufficient evidence cannot establish success.The input `MetricEvidence` carries explicit aggregation semantics, a measured value, and optional activity or denominator counts. Every measurement has an `EvidenceScope` naming the team, relevant population, and the check's existing fixed window. Supporting counts used for qualification must match all three. For property sums, relevant activity counts valid property contributions. A trusted source adapter establishes the population key and relevance; an arbitrary metric label does not. This module neither derives relevance from query text nor executes queries.@@ -180,7 +181,7 @@ The input `MetricEvidence` carries explicit aggregation semantics, a measured va`EvidenceQualification` returns `status` (`sufficient`, `insufficient`, `unsupported`, or `errored`) and a machine-readable `reason`. Missing measurements, supporting-count scope mismatches, and empty windows remain insufficient. Query errors return `errored/query_error`; malformed numeric inputs raise `ValueError` rather than becoming evidence. No qualification result is a passed or failed check verdict.-No interim samples are required. A short, nonempty window can qualify from its final measurement. Historical baselines and interim observations are not inputs. There is no universal denominator minimum, confidence claim, ingestion-lag allowance, or deployment-readiness rule.+No interim samples are required. A short, nonempty window can qualify from its final measurement. Historical baselines and interim observations are not inputs. The count-ratio adapter explicitly supplies a minimum denominator of one; this does not establish a minimum for averages or other adapters. There is no confidence claim, ingestion-lag allowance, or deployment-readiness rule.Final evaluator integration must enforce the organization-level `signals-report-monitoring` gate and its fail-closed Resolved fallback. Source adapters must validate aggregation semantics, preserve the population and fixed window, collect required supporting counts, and establish data readiness. Supported property sums exclude formulas and session-level aggregations. Arbitrary formulas, distinct counts, and quantiles need their own adapter/policy. `MetricMeasurement` supplies neither activity counts nor denominators, and report-metric `minimum_data_points`/`eligibility_query` fields are not inferred as check policy. Final evaluators must bind evidence to the current attempt/check version, compare only sufficient measurements, and separately handle insufficient evidence, unsupported measurements, and query errors. Scheduling, retries, persistence, and report transitions remain outside this contract.diff --git a/products/signals/backend/report_check_metric_evaluation.py b/products/signals/backend/report_check_metric_evaluation.pyindex b31be2a4eeb7..53aa8895be7a 100644--- a/products/signals/backend/report_check_metric_evaluation.py+++ b/products/signals/backend/report_check_metric_evaluation.py@@ -1,7 +1,9 @@from __future__ import annotations+import astimport jsonimport hashlib+from copy import deepcopyfrom dataclasses import replacefrom datetime import datetimefrom uuid import UUID@@ -13,6 +15,7 @@ from posthog.clickhouse.workload import Workloadfrom products.signals.backend.report_check_evaluation import evaluate_check_valuefrom products.signals.backend.report_check_evidence import (+ EvidenceCount,EvidenceScope,EvidenceSource,MetricEvidence,@@ -32,11 +35,59 @@ from products.signals.backend.report_monitoring_results import MonitoringEvaluatlogger = structlog.get_logger(__name__)+def _metric_formula(config: MetricThresholdConfig) -> str | None:+ assert config.query is not None+ trends_filter = config.query["source"].get("trendsFilter") or {}+ if nodes := trends_filter.get("formulaNodes"):+ formula = nodes[0]["formula"] if len(nodes) == 1 else None+ elif formulas := trends_filter.get("formulas"):+ formula = formulas[0] if len(formulas) == 1 else None+ else:+ formula = trends_filter.get("formula")+ return formula if isinstance(formula, str) else None+++def _rate_denominator_index(config: MetricThresholdConfig) -> int | None:+ assert config.query is not None+ series = config.query["source"]["series"]+ if len(series) != 2 or any(+ entity.get("math") not in (None, "total")+ or entity.get("math_property") == "$session_duration"+ or entity.get("math_property_type") == "session_properties"+ for entity in series+ ):+ return None+ formula = _metric_formula(config)+ if formula is None:+ return None+ tree = ast.parse(formula.strip().lower())+ if len(tree.body) != 1 or not isinstance(tree.body[0], ast.Expr):+ return None+ expression = tree.body[0].value+ match expression:+ case (+ ast.BinOp(left=ratio, op=ast.Mult(), right=ast.Constant(value=100))+ | ast.BinOp(left=ast.Constant(value=100), op=ast.Mult(), right=ratio)+ ):+ expression = ratio+ match expression:+ case ast.BinOp(left=ast.Name(id=numerator), op=ast.Div(), right=ast.Name(id=denominator)) if {+ numerator,+ denominator,+ } == {"a", "b"}:+ return ord(denominator) - ord("a")+ return None++def metric_evidence_source(config: MetricThresholdConfig) -> EvidenceSource:assert config.query is not Nonesource = config.query["source"]series = source["series"]- if len(series) != 1 or source.get("samplingFactor") not in (None, 1):+ if source.get("samplingFactor") not in (None, 1):+ return "unsupported"+ if _rate_denominator_index(config) is not None:+ return "rate"+ if len(series) != 1:return "unsupported"if any((source.get("trendsFilter") or {}).get(key) for key in ("formula", "formulas", "formulaNodes")):return "unsupported"@@ -120,8 +171,23 @@ def evaluate_final_metric()assert config.query is not Nonequery = monitoring_metric_query(config.query, start=member.window_start, cutoff=member.window_end)- query_hash = hashlib.sha256(json.dumps(query, sort_keys=True, separators=(",", ":")).encode()).hexdigest()- references = (f"signal-report-check:{check.id}:{member.definition_version}", f"trends-query:sha256:{query_hash}")+ queries = [query]+ formula = None+ denominator_index = None+ if source == "rate":+ denominator_index = _rate_denominator_index(config)+ assert denominator_index is not None+ formula = _metric_formula(config)+ query = deepcopy(query)+ for key in ("formula", "formulas", "formulaNodes"):+ query["source"]["trendsFilter"].pop(key, None)+ queries.append(query)+ query_references = (+ "trends-query:sha256:"+ + hashlib.sha256(json.dumps(item, sort_keys=True, separators=(",", ":")).encode()).hexdigest()+ for item in queries+ )+ references = (f"signal-report-check:{check.id}:{member.definition_version}", *query_references)try:measurement = measure_metric(query,@@ -131,6 +197,7 @@ def evaluate_final_metric(require_full_timeout=True,ch_user=ClickHouseUser.SIGNALS,workload=Workload.OFFLINE,+ formula=formula,)if measurement.measured_at < member.window_end:return replace(@@ -141,7 +208,16 @@ def evaluate_final_metric(explanation="The query result predates the end of the observation window. A fresh result is needed.",evidence_refs=references,)- qualification = qualify_metric_evidence(MetricEvidence(scope=scope, source=source, value=measurement.value))+ denominator = None+ if denominator_index is not None:+ counts = measurement.source_values+ if counts is None or len(counts) != 2 or any(count < 0 or count % 1 != 0 for count in counts):+ raise ValueError("A rate needs two non-negative integer counts.")+ denominator = EvidenceCount(scope=scope, value=int(counts[denominator_index]))+ qualification = qualify_metric_evidence(+ MetricEvidence(scope=scope, source=source, value=measurement.value, denominator=denominator),+ minimum_denominator=1 if source == "rate" else None,+ )comparison = monitoring_comparison(config, member.window_end - member.window_start)verdict = evaluate_check_value(comparison=comparison, observed_value=measurement.value, subject=check.title)except MetricQueryBudgetExhausted:@@ -157,10 +233,14 @@ def evaluate_final_metric(evidence_refs=references,)explanation = f"{verdict.explanation}. Query measured at {measurement.measured_at.isoformat()}."- if qualification.status == "insufficient":+ if qualification.reason == "empty_denominator":+ explanation = f"The final value was {measurement.value}, measured at {measurement.measured_at.isoformat()}. The rate's denominator was zero in the observation window; the value cannot establish success or failure."+ elif qualification.status == "insufficient":explanation = f"The final value was {measurement.value}, measured at {measurement.measured_at.isoformat()}. Relevant activity for this metric's population has not been established; the value cannot establish success or failure."elif qualification.status == "unsupported":explanation = f"The final value was {measurement.value}, measured at {measurement.measured_at.isoformat()}. This value cannot establish success or failure under the metric's evidence policy."+ if denominator is not None:+ explanation += f" Denominator: {denominator.value}, measured at {measurement.measured_at.isoformat()}."return replace(evaluation,evidence_status=qualification.status,diff --git a/products/signals/backend/report_metric_refresh.py b/products/signals/backend/report_metric_refresh.pyindex 70a1fac203fe..a0e4f86a7be9 100644--- a/products/signals/backend/report_metric_refresh.py+++ b/products/signals/backend/report_metric_refresh.py@@ -33,6 +33,7 @@ from posthog.clickhouse.client.connection import ClickHouseUserfrom posthog.clickhouse.query_tagging import tag_queriesfrom posthog.clickhouse.workload import Workloadfrom posthog.dataclasses import frozen+from posthog.hogql_queries.utils.formula_ast import FormulaASTfrom posthog.models import Teamfrom products.product_analytics.backend.facade.queries import run_cached_trends_query@@ -65,6 +66,7 @@ class MetricMeasurement:value: floatmeasured_at: datetimeseries: list[float] | None+ source_values: list[float] | None = None@frozen@@ -115,8 +117,8 @@ def _run_metric_query(require_full_timeout: bool = False,ch_user: ClickHouseUser = ClickHouseUser.DEFAULT,workload: Workload = Workload.DEFAULT,-) -> tuple[dict, datetime | None]:- """Run the stored query in one derived display shape and return its first series and response.+) -> tuple[list[dict[str, object]], datetime | None]:+ """Run the stored query in one derived display shape and return its series and response.The derived shapes match the ones the report detail sends through the frontend Query path, soboth sides read and warm the same cache entries.@@ -154,9 +156,9 @@ def _run_metric_query(workload=workload,)results = response.results- if not isinstance(results, list) or not results or not isinstance(results[0], dict):+ if not isinstance(results, list) or not results or any(not isinstance(result, dict) for result in results):raise ValueError("metric query returned no series")- return results[0], response.last_refresh+ return results, response.last_refreshdef _finite_number(raw: object, *, what: str) -> float:@@ -168,7 +170,7 @@ def _finite_number(raw: object, *, what: str) -> float:return value-def whole_window_value(+def whole_window_values(query: dict[str, Any],team: Team,*,@@ -176,8 +178,8 @@ def whole_window_value(require_full_timeout: bool = False,ch_user: ClickHouseUser = ClickHouseUser.DEFAULT,workload: Workload = Workload.DEFAULT,-) -> tuple[float, datetime]:- first_series, last_refresh = _run_metric_query(+) -> tuple[list[float], datetime]:+ results, last_refresh = _run_metric_query(query,team,ChartDisplayType.BOLD_NUMBER,@@ -186,11 +188,31 @@ def whole_window_value(ch_user=ch_user,workload=workload,)- value = _finite_number(first_series.get("aggregated_value"), what="aggregate")+ values = [_finite_number(result.get("aggregated_value"), what="aggregate") for result in results]measured_at = last_refresh if isinstance(last_refresh, datetime) else timezone.now()if measured_at.tzinfo is None or measured_at.utcoffset() is None:measured_at = measured_at.replace(tzinfo=UTC)- return value, measured_at+ return values, measured_at+++def whole_window_value(+ query: dict[str, Any],+ team: Team,+ *,+ deadline: float = math.inf,+ require_full_timeout: bool = False,+ ch_user: ClickHouseUser = ClickHouseUser.DEFAULT,+ workload: Workload = Workload.DEFAULT,+) -> tuple[float, datetime]:+ values, measured_at = whole_window_values(+ query,+ team,+ deadline=deadline,+ require_full_timeout=require_full_timeout,+ ch_user=ch_user,+ workload=workload,+ )+ return values[0], measured_atdef longitudinal_values(@@ -203,10 +225,10 @@ def longitudinal_values() -> list[float]:"""The trailing per-bucket values, oldest first, for the row-sized trend strip."""- first_series, _ = _run_metric_query(+ results, _ = _run_metric_query(query, team, ChartDisplayType.ACTIONS_BAR, deadline=deadline, ch_user=ch_user, workload=workload)- raw_points = first_series.get("data")+ raw_points = results[0].get("data")if not isinstance(raw_points, list):raise ValueError("metric query returned no buckets")return [_finite_number(point, what="bucket") for point in raw_points[-MAX_METRIC_SERIES_POINTS:]]@@ -221,7 +243,24 @@ def measure_metric(require_full_timeout: bool = False,ch_user: ClickHouseUser = ClickHouseUser.DEFAULT,workload: Workload = Workload.DEFAULT,+ formula: str | None = None,) -> MetricMeasurement:+ """An explicit formula consumes raw source series and omits bucket series.++ Keeping its value and source values in one cached response prevents evidence from+ qualifying a formula calculated from a different snapshot.+ """+ if formula is not None:+ source_values, measured_at = whole_window_values(+ query,+ team,+ deadline=deadline,+ require_full_timeout=require_full_timeout,+ ch_user=ch_user,+ workload=workload,+ )+ value = _finite_number(FormulaAST([[item] for item in source_values]).call(formula)[0], what="formula")+ return MetricMeasurement(value=value, measured_at=measured_at, series=None, source_values=source_values)value, measured_at = whole_window_value(query,team,diff --git a/products/signals/backend/test/test_report_check_final_metrics.py b/products/signals/backend/test/test_report_check_final_metrics.pyindex 1a868e5620f2..ea0b5cb9b5f9 100644--- a/products/signals/backend/test/test_report_check_final_metrics.py+++ b/products/signals/backend/test/test_report_check_final_metrics.py@@ -3,7 +3,7 @@ from datetime import timedeltafrom uuid import uuid4import time_machine-from posthog.test.base import BaseTest, _create_event+from posthog.test.base import BaseTest, _create_event, flush_persons_and_eventsfrom unittest.mock import patchfrom django.test import SimpleTestCase@@ -15,6 +15,7 @@ from posthog.clickhouse.client.connection import ClickHouseUserfrom posthog.clickhouse.workload import Workloadfrom posthog.models import Organization, Team+from products.product_analytics.backend.facade.contracts import TrendsQueryRunResultfrom products.signals.backend.models import (SignalReport,SignalReportCheck,@@ -35,6 +36,40 @@ _FLAG = "products.signals.backend.report_content_gates.feature_enabled_or_false"class TestMetricEvidenceAdapter(SimpleTestCase):+ @parameterized.expand(+ [+ ("ratio", "formula", "A / B", "total", "rate"),+ ("trailing_semicolon", "formula", "A / B;", "total", "rate"),+ ("percentage", "formulas", "(A / B) * 100", "total", "rate"),+ ("reverse_percentage", "formulaNodes", "100 * (B / A)", "total", "rate"),+ ("sum_ratio", "formula", "A / B", "sum", "unsupported"),+ ("distinct_ratio", "formula", "A / B", "dau", "unsupported"),+ ("difference", "formula", "A - B", "total", "unsupported"),+ ("self_ratio", "formula", "A / A", "total", "unsupported"),+ ("compound_denominator", "formula", "A / (A + B)", "total", "unsupported"),+ ]+ )+ def test_only_count_ratios_have_a_denominator_adapter(+ self, _name: str, formula_key: str, formula: str, math: str, expected: str+ ) -> None:+ query = trends_metric_query(+ series=[+ {"kind": "EventsNode", "event": "error", "math": math, "math_property": "amount"},+ {"kind": "EventsNode", "event": "attempt", "math": "total"},+ ]+ )+ query["source"]["trendsFilter"] = {+ formula_key: [{"formula": formula}]+ if formula_key == "formulaNodes"+ else [formula]+ if formula_key == "formulas"+ else formula+ }+ if formula_key != "formula":+ query["source"]["trendsFilter"]["formula"] = "A - B"+ config = MetricThresholdConfig.model_validate({"query": query, "comparison": {"operator": "lte", "value": 1}})+ assert metric_evidence_source(config) == expected+@parameterized.expand([("default_count", None, None, "count"),@@ -105,6 +140,7 @@ class TestFinalMetricExecution(BaseTest):formula: bool = False,scaling: str = "fixed",event: str = "checkout",+ rate_formula: str | None = None,) -> FinalCheck:config = {"query": trends_metric_query(@@ -118,6 +154,15 @@ class TestFinalMetricExecution(BaseTest):config["monitoring"] = {"reference_period_minutes": 120, "threshold_scaling": scaling}if formula:config["query"]["source"]["trendsFilter"] = {"formula": "A * 2"}+ if rate_formula is not None:+ config["query"]["source"]["series"].append(+ {"kind": "EventsNode", "event": f"{event}_attempt", "math": "total"}+ )+ config["query"]["source"]["properties"] = [+ {"key": "environment", "value": ["production"], "operator": "exact", "type": "event"}+ ]+ config["query"]["source"]["trendsFilter"] = {"formula": rate_formula}+ config["comparison"] = {"operator": "lte", "value": 2 if "100" in rate_formula else 0.02}check = SignalReportCheck.objects.for_team(self.team.id).create(team_id=self.team.id,report=self.report,@@ -143,7 +188,7 @@ class TestFinalMetricExecution(BaseTest):)def _run(self) -> None:- run_due_final_metrics(now=timezone.now(), deadline=time.monotonic() + 30, limit=50)+ run_due_final_metrics(now=timezone.now(), deadline=time.monotonic() + 120, limit=50)@parameterized.expand([@@ -153,16 +198,34 @@ class TestFinalMetricExecution(BaseTest):("zero_count", "total", 0, "inconclusive"),("zero_sum", "sum", 0, "inconclusive"),("negative_sum", "sum", -1, "inconclusive"),+ ("rate_pass", "total", 0.01, "passed", "A / B", 100),+ ("terminated_rate", "total", 0.01, "passed", "A / B;", 100),+ ("rate_fail", "total", 0.03, "failed", "A / B", 100),+ ("zero_rate_with_activity", "total", 0, "passed", "A / B", 1),+ ("zero_rate_without_activity", "total", 0, "inconclusive", "A / B", 0),+ ("percentage_pass", "total", 1, "passed", "A / B * 100", 100),+ ("reverse_rate", "total", 0.01, "passed", "B / A", 100),])def test_final_results_separate_evidence_from_matching_and_aggregate_report_outcomes(- self, _name: str, math: str, value: float, outcome: str+ self,+ _name: str,+ math: str,+ value: float,+ outcome: str,+ rate_formula: str | None = None,+ denominator: int | None = None,) -> None:- member = self._member(math=math)+ member = self._member(math=math, rate_formula=rate_formula)+ source_values = None+ if denominator is not None:+ numerator = value * denominator / (100 if rate_formula and "100" in rate_formula else 1)+ source_values = [denominator, numerator] if rate_formula == "B / A" else [numerator, denominator]+ measurement = MetricMeasurement(value=value, measured_at=self.now, series=None, source_values=source_values)with (patch("products.signals.backend.report_check_telemetry.ph_background_capture") as capture,self.captureOnCommitCallbacks(execute=True),- patch(_MEASURE, return_value=MetricMeasurement(value=value, measured_at=self.now, series=None)) as measure,+ patch(_MEASURE, return_value=measurement) as measure,):summary = run_due_report_checks(now=self.now)self._run()@@ -175,8 +238,8 @@ class TestFinalMetricExecution(BaseTest):assert result.window_start == member.window_start and result.window_end == member.window_endassert result.definition_version == member.definition_versionassert str(value) in result.explanation- assert len(result.evidence_refs) == 2- query = measure.call_args.args[0]+ assert len(result.evidence_refs) == (3 if rate_formula else 2)+ query = measure.call_args_list[0].args[0]assert query["source"]["dateRange"] == {"date_from": member.window_start.isoformat(),"date_to": member.window_end.isoformat(),@@ -186,6 +249,11 @@ class TestFinalMetricExecution(BaseTest):assert measure.call_args.kwargs["workload"] == Workload.OFFLINEassert measure.call_args.kwargs["ch_user"] == ClickHouseUser.SIGNALSassert measure.call_count == 1+ if rate_formula:+ assert len(query["source"]["series"]) == 2+ assert "formula" not in query["source"]["trendsFilter"]+ assert measure.call_args.kwargs["formula"] == rate_formula+ assert member.check.config["query"]["source"]["trendsFilter"]["formula"] == rate_formulaassert sum([summary.passed, summary.failed, summary.inconclusive]) == 1self.report.refresh_from_db()member.check.refresh_from_db()@@ -196,7 +264,9 @@ class TestFinalMetricExecution(BaseTest):assert self.report.status == expected_statusassert member.check.runs_remaining == 3assert member.check.last_outcome == (outcome if outcome in {"passed", "failed"} else None)- if value == 0:+ if denominator == 0:+ assert result.evidence_status == "insufficient" and result.evidence_reason == "empty_denominator"+ elif value == 0 and rate_formula is None:assert result.evidence_status == "insufficient" and result.evidence_reason == "activity_unavailable"elif value < 0:assert result.evidence_status == "unsupported"@@ -241,15 +311,27 @@ class TestFinalMetricExecution(BaseTest):)measure.assert_not_called()- @parameterized.expand([("exhausted", False), ("recovered", True)])+ @parameterized.expand(+ [+ ("exhausted", False),+ ("recovered", True),+ ("rate_exhausted", False, "A / B"),+ ("rate_recovered", True, "A / B"),+ ]+ )def test_query_errors_retry_three_times_on_fixed_windows_and_remain_durable(- self, _name: str, recovered: bool+ self, _name: str, recovered: bool, rate_formula: str | None = None) -> None:- member = self._member()+ member = self._member(rate_formula=rate_formula)responses = [RuntimeError("private query detail"),RuntimeError("private query detail"),- MetricMeasurement(value=12, measured_at=self.now, series=None)+ MetricMeasurement(+ value=0 if rate_formula else 12,+ measured_at=self.now,+ series=None,+ source_values=[0, 12] if rate_formula else None,+ )if recoveredelse RuntimeError("private query detail"),]@@ -273,20 +355,29 @@ class TestFinalMetricExecution(BaseTest):self._run()assert SignalReportMonitoringResult.objects.for_team(self.team.id).get().outcome == outcome- def test_cached_predeadline_results_cannot_establish_a_final_pass(self) -> None:- member = self._member()+ @parameterized.expand([("metric",), ("rate_inputs",)])+ def test_cached_predeadline_results_cannot_establish_a_final_pass(self, stale: str) -> None:+ member = self._member(rate_formula="A / B" if stale == "rate_inputs" else None)+ measurement = MetricMeasurement(+ value=0 if stale == "rate_inputs" else 12,+ measured_at=member.window_end - timedelta(seconds=1),+ series=None,+ source_values=[0, 12] if stale == "rate_inputs" else None,+ )with patch(_MEASURE,- return_value=MetricMeasurement(value=12, measured_at=member.window_end - timedelta(seconds=1), series=None),+ return_value=measurement,):self._run()result = SignalReportMonitoringResult.objects.for_team(self.team.id).get()assert result.evidence_status == "insufficient" and result.expectation_matches is None- @parameterized.expand([("total", 3), ("sum", 6)])- def test_real_query_excludes_events_outside_the_fixed_window_and_population(self, math: str, value: int) -> None:+ @parameterized.expand([("total", 3), ("sum", 6), ("total", 0.03, "A / B"), ("total", 3, "A / B * 100")])+ def test_real_query_excludes_events_outside_the_fixed_window_and_population(+ self, math: str, value: float, rate_formula: str | None = None+ ) -> None:population = f"checkout-{uuid4()}"- member = self._member(math=math, event=population)+ member = self._member(math=math, event=population, rate_formula=rate_formula)for timestamp, event, count in [(member.window_start - timedelta(seconds=1), population, 20),(member.window_start + timedelta(minutes=1), population, 3),@@ -295,20 +386,73 @@ class TestFinalMetricExecution(BaseTest):]:for _ in range(count):_create_event(- team=self.team, event=event, distinct_id="test-user", timestamp=timestamp, properties={"amount": 2}+ team=self.team,+ event=event,+ distinct_id="test-user",+ timestamp=timestamp,+ properties={"amount": 2, "environment": "production"},)+ if rate_formula:+ for timestamp, environment, count in [+ (member.window_start + timedelta(minutes=1), "production", 100),+ (member.window_start + timedelta(minutes=1), "test", 100),+ (member.window_start - timedelta(seconds=1), "production", 100),+ (member.window_end + timedelta(seconds=1), "production", 100),+ ]:+ for _ in range(count):+ _create_event(+ team=self.team,+ event=f"{population}_attempt",+ distinct_id="test-user",+ timestamp=timestamp,+ properties={"environment": environment},+ )+ flush_persons_and_events()self._run()result = SignalReportMonitoringResult.objects.for_team(self.team.id).get()assert result.outcome == "failed" and result.evidence_status == "sufficient", result.explanationassert str(float(value)) in result.explanation+ if rate_formula:+ assert "Denominator: 100" in result.explanation- def test_budget_deferral_does_not_spend_a_technical_retry(self) -> None:- self._member()+ @parameterized.expand([("count", None), ("rate_inputs", "A / B")])+ def test_budget_deferral_does_not_spend_a_technical_retry(self, _name: str, rate_formula: str | None) -> None:+ self._member(rate_formula=rate_formula)with patch(_MEASURE, side_effect=MetricQueryBudgetExhausted):self._run()assert not SignalReportMonitoringResult.objects.for_team(self.team.id).exists()assert len(collect_due_final_metrics(self.now, limit=10)) == 1+ def test_cached_empty_rate_cannot_use_a_newer_denominator_after_budget_deferral(self) -> None:+ self._member(rate_formula="A / B")+ deferred = False++ def query_response(*, query: dict[str, object], **kwargs: object) -> TrendsQueryRunResult:+ nonlocal deferred+ trends_filter = query["trendsFilter"]+ series = query["series"]+ assert isinstance(trends_filter, dict) and isinstance(series, list)+ cached_formula = bool(trends_filter.get("formula"))+ if cached_formula:+ results = [{"aggregated_value": 0}]+ elif not deferred:+ deferred = True+ raise MetricQueryBudgetExhausted+ else:+ results = [{"aggregated_value": 10} for _ in series]+ return TrendsQueryRunResult(results=results, last_refresh=self.now if cached_formula else timezone.now())++ with patch(+ "products.signals.backend.report_metric_refresh.run_cached_trends_query", side_effect=query_response+ ):+ self._run()+ assert not SignalReportMonitoringResult.objects.for_team(self.team.id).exists()+ with time_machine.travel(self.now + timedelta(minutes=1), tick=False):+ self._run()+ result = SignalReportMonitoringResult.objects.for_team(self.team.id).get()+ assert result.outcome == "failed", result.explanation+ assert "Denominator: 10" in result.explanation+@parameterized.expand([("short", 19.9, False), ("full", 20.0, True)])def test_only_queries_with_the_full_timeout_can_spend_a_technical_retry(self, _name: str, remaining_seconds: float, spends_retry: bool