Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(signals): evaluate final monitoring metrics (#123675)

Mikayla Thompson5c2ebd4cdd6b
diff --git a/.github/new-events-schema-targets.txt b/.github/new-events-schema-targets.txt
index 8d26f0eed189..cdbdda95b3a7 100644
--- a/.github/new-events-schema-targets.txt
+++ b/.github/new-events-schema-targets.txt
@@ -47,6 +47,7 @@ products/product_analytics/backend/hogql_queries
products/product_analytics/backend/tests/test_insight_data_checks.py
products/replay_vision/backend/tests/test_session_identity.py
products/signals/backend/test/test_anomaly_search.py
+products/signals/backend/test/test_report_check_final_metrics.py
products/signals/backend/test/test_anomaly_scoring_tick.py
products/signals/backend/test/test_first_report_evals.py
products/web_analytics/backend/hogql_queries
diff --git a/docs/internal/signals-pr-lifecycle.md b/docs/internal/signals-pr-lifecycle.md
index 8f1497c26bd5..72f52476e27c 100644
--- a/docs/internal/signals-pr-lifecycle.md
+++ b/docs/internal/signals-pr-lifecycle.md
@@ -18,7 +18,7 @@ The backend `complete_implementation` service records a durable attempt and free
`save_monitoring_result(MonitoringEvaluation(...))` stores final evaluation submissions separately from interim observations and activity artefacts. Each submission identifies the environment, report, attempt, check, definition version, fixed evidence window, evaluation time, qualification status and reason, expectation match, explanation, and opaque evidence references. Evaluators supply qualification using their own evidence policy; metric evaluators can use `qualify_metric_evidence`. Only sufficient evidence can produce a pass or failure. Insufficient or unsupported evidence produces an inconclusive outcome; execution errors remain errors.
-The save transaction locks the report before the check. Replaying an identical submission ID returns its saved row even after lifecycle changes; changing its content raises `MonitoringResultConflict`. New submissions require enabled Monitoring, the current attempt's membership and definition version, an unchanged open and unexpired check, and an explicit completed soak. The evidence window must equal `[monitoring_started_at, monitoring_started_at + soak_minutes]`, including zero soak. New IDs preserve inconclusive and error history and allow later evaluation. Once a pass or failure exists, another submission cannot replace it. Invalidated new work raises `InvalidMonitoringResult`. Evaluators must retain their submission ID and payload across delivery retries and use a fresh ID for a new evaluation. This interface does not run evaluators, schedule retries, spend check runs, complete checks, or transition reports.
+The save transaction locks the report before the check. Replaying an identical submission ID returns its saved row even after lifecycle changes; changing its content raises `MonitoringResultConflict`. New submissions require enabled Monitoring, the current attempt's membership and definition version, an unchanged check, and an explicit completed soak. Pending, active, technically errored, and expired members remain eligible for final evaluation: interim execution errors and safety expiry do not remove a frozen attempt's final obligation. Cancelled or completed checks remain ineligible. Saving a result does not reactivate an errored or expired check. The evidence window must equal `[monitoring_started_at, monitoring_started_at + soak_minutes]`, including zero soak. New IDs preserve inconclusive and error history and allow later evaluation. Once a pass or failure exists, another submission cannot replace it. Invalidated new work raises `InvalidMonitoringResult`. Evaluators must retain their submission ID and payload across delivery retries and use a fresh ID for a new evaluation. This interface does not run evaluators, schedule retries, spend check runs, complete checks, or transition reports.
Monitoring keeps the researched report available for verification and new matching signals, without starting another implementation. Enabled responses expose the server-owned start time, and implementation work is done while the report itself remains unfinished. A Monitoring child keeps its parent plan open, and child rollups cannot finish a Monitoring parent. The single and bulk state APIs accept the Monitoring target. Start monitoring appears after follow-up checks load and at least one exists; the service decides whether any remain applicable when the request runs. Existing Monitoring reports can be finished manually, reopened, or archived. Manual completion records caller attribution and explicitly states that automatic verification and an evidence amendment were not performed. PR merges continue to resolve reports.
@@ -54,7 +54,7 @@ Manual entry calls `complete_implementation` from `SignalReportViewSet._transiti
The service records stale expiries before selecting this report's pending or active, unexpired checks, independently of kind and approval. Historical terminal checks, standalone metrics, and related reports' checks do not qualify. No applicable checks resolve directly without a verified-success claim. Applicable checks enter Monitoring with the current attempt's durable membership and version snapshot. Missing Monitoring configuration or soak blocks verification; zero soak is valid. Carried-over active checks require fresh evidence.
-Archive and restore preserve the attempt and original start, including expiry recovery and saved interim observations. Reopening implementation clears the period; completing implementation again creates a fresh attempt. Disabled or unavailable evaluations refuse new manual entry and use the existing Resolved fallback. Final evaluation, automatic completion, amendments, and PR-merge entry remain separate work.
+Archive and restore preserve the attempt and original start, including expiry recovery and saved interim observations. Reopening implementation clears the period; completing implementation again creates a fresh attempt. Disabled or unavailable evaluations refuse new manual entry and use the existing Resolved fallback. Final metric evaluation records results separately from manual completion. Automatic completion, amendments, and PR-merge entry remain separate work.
## Follow-up checks
@@ -70,7 +70,7 @@ Report reads, research selection, and snapshot refresh use the same filter check
Snapshot refresh still requires access to both the live query and the shared measurement.
A viewer with a property grant can read the live query while the team's default property restrictions keep the shared snapshot hidden.
-Research writes measurable outcome goals as `metric_threshold` report checks and investigative goals as `agent` checks. A metric check stores a bounded live query, baseline, comparison, and soak window; the query and display format are copied from its report metric when it names one. Checks execute only during enabled Monitoring. The coordinator records interim metric observations within each check's soak window; final metric and authored agent verification remain separate work. Resolving a report stops execution, including recurrence checks. Historical results remain readable. New metric checks validate count, rate, duration, baseline, and threshold values using the same rules as report metrics; existing configurations remain readable. A later research pass reviews every open check, preserves unchanged checks and their approvals, replaces changed checks, and retires omitted checks. A failed verification turn leaves existing checks alone. Verification can revise only Expected impact; other summary sections and their chart links must remain intact. The full replacement schedule, including its soak and measurement window, must finish before the 90-day horizon. The Follow-up checks sidebar shows schedules and results for both kinds. The Expected impact section uses the same metric checks to show goals and charts, behind the person-level `signals-expected-impact` display flag. A person can mark those measurements "Looks good" as a quality signal; approval does not control scheduling or execution. The "Suggest different metrics" action starts a discussion that can atomically replace relevant open metric checks while preserving unrelated checks. A failed replacement leaves the original running, and a successful replacement starts unapproved. Report observation metrics remain separate from these forward-looking checks.
+Research writes measurable outcome goals as `metric_threshold` report checks and investigative goals as `agent` checks. A metric check stores a bounded live query, baseline, comparison, and soak window; the query and display format are copied from its report metric when it names one. Checks execute only during enabled Monitoring. The coordinator records interim metric observations within each check's soak window; final metric evaluation records fixed-window results after each deadline. Authored-agent verification remains separate work. Resolving a report stops execution, including recurrence checks. Historical results remain readable. New metric checks validate count, rate, duration, baseline, and threshold values using the same rules as report metrics; existing configurations remain readable. A later research pass reviews every open check, preserves unchanged checks and their approvals, replaces changed checks, and retires omitted checks. A failed verification turn leaves existing checks alone. Verification can revise only Expected impact; other summary sections and their chart links must remain intact. The full replacement schedule, including its soak and measurement window, must finish before the 90-day horizon. The Follow-up checks sidebar shows schedules and results for both kinds. The Expected impact section uses the same metric checks to show goals and charts, behind the person-level `signals-expected-impact` display flag. A person can mark those measurements "Looks good" as a quality signal; approval does not control scheduling or execution. The "Suggest different metrics" action starts a discussion that can atomically replace relevant open metric checks while preserving unrelated checks. A failed replacement leaves the original running, and a successful replacement starts unapproved. Report observation metrics remain separate from these forward-looking checks.
Reports awaiting human input retain their generated checks for a later Monitoring period. Outside Monitoring, open checks display as paused in the Follow-up checks sidebar. Expected impact retains proposed or approved measurement labels before Monitoring, and shows paused checks on resolved or archived reports, including when the rollout is disabled. Stored schedules do not advertise future runs or active scouts. Historical verdicts and legacy schedule fields remain readable; remaining-run counts do not authorize post-resolution execution. Metric replacements require access to the query they schedule and preserve the remaining recurring runs and soak duration, including zero minutes. Replacement cannot override the soak. Creation and replacement share full schedule validation against the 90-day horizon. A replacement rejects a check moved by a concurrent report merge; reload the report and retry on the survivor. Units cannot contain null characters or unpaired Unicode surrogates. The Expected impact section shows finished verdicts and refreshes after agent tasks change checks.
@@ -90,7 +90,7 @@ The report detail action is **Start monitoring**, with the tooltip “PostHog wa
Open checks on Monitoring reports expose a server-derived `monitoring_timing` field when `signals-report-monitoring` is enabled. The field separates a scheduled or due interim measurement from the end of that check's soak window, and identifies paused, archived, unarmed, expired, or incomplete timing. It is omitted for other checks and when the flag is disabled or unavailable. Timing reads do not arm checks, change their schedules, or decide outcomes.
-The Follow-up checks rail and Expected impact details show the next interim measurement separately from the check's observation-window end. A closed window reads as awaiting verification, without advertising its deadline as another run. Interim measurements do not spend remaining verification runs. Agent checks and metric checks without a Monitoring definition remain paused during Monitoring. Final evaluation and automatic report completion remain separate work.
+The Follow-up checks rail and Expected impact details show the next interim measurement separately from the check's observation-window end. A closed window reads as awaiting verification, without advertising its deadline as another run. Interim measurements do not spend remaining verification runs. Agent checks and metric checks without a Monitoring definition remain paused during Monitoring. Final metric results are stored independently of this timing presentation. Automatic report completion remains separate work.
Open report details refresh check timing every minute and reload checks immediately when the report status changes, so resolution or archiving replaces cached Monitoring timing before polling stops.
@@ -98,16 +98,33 @@ Research should make every effort to complete `config.monitoring` for each metri
The definition describes a measurement from Monitoring start through the observation time. A `linear` target scales by elapsed minutes divided by reference minutes and requires a single additive event count or property sum without a formula. Session-level sums are excluded because a session can contribute to more than one chart bucket. A `fixed` target keeps its bound as the observation period grows, including for percentages, averages, distinct counts, or explicitly fixed custom targets. Rolling weekly and monthly active-user queries are refused because their lookback overrides the Monitoring bounds; use a distinct user count instead. Display format alone never selects the policy. The baseline retains its own measured period and is not presented as an observation from Monitoring.
-Expected impact keeps the activity chart visible and collapses the target period, scaling policy, baseline, and check timing under measurement details. Dynamic chart boundaries refresh at minute intervals so repeated views can reuse cached results. Bucket granularity follows the full displayed range and stays within the 1,000-point budget. The chart preserves date-range filters and spans one reference period on each side of Monitoring start, or now before Monitoring starts, extending through now if necessary. Start and now markers separate the surrounding activity from future dates, whose values remain empty. Compatible charts show target reference lines: additive linear targets scale to a full chart bucket, and fixed rates or property averages/quantiles retain their level. Fixed totals, distinct counts, unknown aggregations, linear calendar intervals longer than a day, and linear day buckets with unequal daylight-saving durations receive no reference line. These references do not impose per-bucket pass/fail rules. This read-time chart does not evaluate the Monitoring target or save a check result. The stored bounded query and whole-window comparison remain available for later final verification. Final Monitoring outcomes remain separate work.
+Expected impact keeps the activity chart visible and collapses the target period, scaling policy, baseline, and check timing under measurement details. Dynamic chart boundaries refresh at minute intervals so repeated views can reuse cached results. Bucket granularity follows the full displayed range and stays within the 1,000-point budget. The chart preserves date-range filters and spans one reference period on each side of Monitoring start, or now before Monitoring starts, extending through now if necessary. Start and now markers separate the surrounding activity from future dates, whose values remain empty. Compatible charts show target reference lines: additive linear targets scale to a full chart bucket, and fixed rates or property averages/quantiles retain their level. Fixed totals, distinct counts, unknown aggregations, linear calendar intervals longer than a day, and linear day buckets with unequal daylight-saving durations receive no reference line. These references do not impose per-bucket pass/fail rules. This read-time chart does not evaluate the Monitoring target or save a check result. Final metric verification uses the stored bounded query and authored comparison independently of the chart. Final Monitoring outcomes remain separate work.
-During Monitoring, checks with an explicit Monitoring definition sample at their configured run interval, or every 24 hours when none is set. The first observation is due one interval after Monitoring starts. Interim observations run only before Monitoring start plus the check's `soak_minutes`. The full trailing query window does not delay them. Interim queries explicitly use the offline ClickHouse workload; interactive metric refreshes retain their default workload. The next due time is capped at that soak deadline, but this PR does not execute a final check or conclude a report there. A missing or zero soak, or a soak no longer than the sampling interval, produces no interim observations. Each query pins both bounds to Monitoring start and one captured observation cutoff, preserving the population filters. Pinned ranges include partial periods so their bounds match the scaled target. Linear comparisons scale both bounds by elapsed time divided by the reference period, without rounding; fixed comparisons stay unchanged.
+During Monitoring, checks with an explicit Monitoring definition sample at their configured run interval, or every 24 hours when none is set. The first observation is due one interval after Monitoring starts. Interim observations run only before Monitoring start plus the check's `soak_minutes`. The full trailing query window does not delay them. Interim queries explicitly use the offline ClickHouse workload; interactive metric refreshes retain their default workload. The next interim due time is capped at that soak deadline. Final metric evaluation then uses its separate execution path; neither path concludes a report. A missing or zero soak, or a soak no longer than the sampling interval, produces no interim observations. Each query pins both bounds to Monitoring start and one captured observation cutoff, preserving the population filters. Pinned ranges include partial periods so their bounds match the scaled target. Linear comparisons scale both bounds by elapsed time divided by the reference period, without rounding; fixed comparisons stay unchanged.
Each completed attempt appends a protected `check_observation` log entry with its exact bounds, soak deadline, display format, unit, effective comparison, value, and match result, or a sanitized query error. The log labels it as an interim measurement, shows whether the goal is on track, and displays time remaining when the measurement was taken. It does not assign a terminal outcome. Observations do not spend `runs_remaining`, change the report status, or emit recurrence signals. The next sample is due one interval after the cutoff; missed intervals do not create catch-up runs. Restoring Monitoring schedules from the latest attempt in that period, preserving a failed attempt's error streak and six-hour retry across suppression. Re-enabling the flag restores the measurement origin, the next attempt from the latest run, and the expiry from the original Monitoring start. This repair precedes the expiry sweep so a fallback expiry cannot shorten or extend the Monitoring horizon. A check that cannot fit its source queries into the remaining coordinator budget stays due without an observation or an error-budget change. Query failures retain the existing six-hour retry and consecutive-error limit, and the 90-day check horizon still applies. Writes reject a changed check, Monitoring period, report state, or flag, and duplicate workers cannot save the same attempt twice. Expiry sweeps and schedule repairs retain their check-row concurrency guards after database lock waits, so they cannot overwrite a concurrent cancellation, expiry extension, or saved observation.
-With Monitoring disabled, observations disappear from the artefact API and counts. No Resolved executor runs. The coordinator also leaves disabled or unavailable Monitoring periods dormant. Observation values require the same query permissions as terminal check results. The inbox follows all artefact pages within the current project before deriving task runs, reviewers, and activity so observations cannot displace older report state. Polling waits for an in-flight paginated load to finish. With the Monitoring flag enabled, scouts see metric checks with an explicit definition as scheduled or due inside their Monitoring window; agent checks and metric checks without a definition remain paused pending final verification support. The soak deadline bounds interim execution; no final verdict, reopening, or follow-up report is added here.
+With Monitoring disabled, observations disappear from the artefact API and counts. No Resolved executor runs. The coordinator also leaves disabled or unavailable Monitoring periods dormant. Observation values require the same query permissions as terminal check results. The inbox follows all artefact pages within the current project before deriving task runs, reviewers, and activity so observations cannot displace older report state. Polling waits for an in-flight paginated load to finish. With the Monitoring flag enabled, scouts see metric checks with an explicit definition as scheduled or due inside their Monitoring window; agent checks and metric checks without a definition remain paused in this presentation. The soak deadline bounds interim execution. Final metric results are saved separately without reopening reports or creating follow-up reports.
Disabling Monitoring stops execution without starting a post-resolution schedule or recalculating a recurring deadline. Re-enabling restores the original Monitoring horizon for active checks and leaves terminal checks unchanged. Pending checks arm only during enabled Monitoring, using the original implementation-complete timestamp. Queued legacy check-triggered scout activities skip before starting agent work, and already-running legacy scouts cannot record a result through the check-result tool. Scout check lists mark these checks paused and do not advertise them as waiting on a run.
+After each metric check's soak ends, the coordinator evaluates its original Monitoring-start-to-deadline window, even if execution runs late.
+Final work uses the current attempt's check membership and definition versions, independently of interim observations and recurring run budgets.
+Candidate collection scans past stale definitions until it fills the eligible batch or reaches the tick deadline.
+Single-source raw event/action counts and event-property sums have supported evidence adapters; sampled values, formulas, distinct counts, and session/property aggregates without an adapter produce explicit unsupported results.
+Session aggregation remains unsupported when a count series carries `$session_duration` or a `session_properties` property type.
+An average without a source-specific minimum denominator produces insufficient evidence.
+Positive counts and supported sums qualify as sufficient evidence; zero totals remain insufficient when no relevant activity source is defined, even if they meet the numeric target.
+A zero-length window cannot establish success or failure, and a cached measurement from before the deadline cannot qualify as final evidence.
+
+Final metric evaluations save through `save_monitoring_result` and retain the definition version, exact window, explanation, measured value and time, and query fingerprint.
+The coordinator makes at most three technical query attempts, retrying five minutes after the first error and fifteen minutes after the second.
+Each retry uses the original window; competing ticks use the same submission ID for the same logical attempt.
+Insufficient and unsupported results settle this evaluator's work. Final queries defer until the tick can grant their full 20-second timeout; budget deferrals do not consume a retry.
+Interim query exhaustion and schedule expiry do not consume the final budget: frozen members remain eligible for final verification without changing their check status, window, or error history.
+State, flag, definition, and attempt guards apply before querying and again when saving.
+Final results leave reports in Monitoring; authored-agent checks, report outcome aggregation, assessment, and follow-up research have separate execution paths.
+
With `signals-report-metrics` enabled, research gathers short-window baselines before verification and prefers metric-based follow-up checks. It aims for practical 70-80% confidence in improvement, without claiming a calculated confidence score. Observed volume, recurrence, ingestion delay, and deployment timing justify the shortest useful query and wait. Frequent or high-volume checks can use 4-8 hours, allowing for several hours of deployment delay before the fix reaches users. Around 24 hours or several days require the corresponding activity or rollout evidence. Agent checks remain available when interpretation or unsupported measurements prevent a useful metric check. Missing activity is not success. These prompt preferences do not change execution, schema defaults, or final outcomes.
The server refuses new Monitoring definitions when `signals-report-monitoring` is disabled or unavailable and omits the definition from check responses. Stored definitions remain intact. Research carries its definition-visibility decision into reconciliation, so flag recovery during a run does not replace an unchanged check or reset its approval and schedule. Existing definitions, charts, check schedules, and Resolved outcomes otherwise follow their current behavior.
diff --git a/products/signals/backend/report_check_execution.py b/products/signals/backend/report_check_execution.py
index 930add694199..991fa371d355 100644
--- a/products/signals/backend/report_check_execution.py
+++ b/products/signals/backend/report_check_execution.py
@@ -1,6 +1,6 @@
-"""Coordinate interim check measurements during enabled Monitoring.
+"""Coordinate interim measurements and final metric verification during enabled Monitoring.
-Final verification and authored agent checks use separate execution contracts. The legacy verdict
+Authored agent checks use a separate execution contract. The legacy verdict
utilities below do not run from the coordinator.
"""
@@ -48,6 +48,7 @@ from products.signals.backend.report_check_evaluation import (
errored_check_explanation,
evaluate_check_value,
)
+from products.signals.backend.report_check_final_metrics import run_due_final_metrics
from products.signals.backend.report_check_monitoring import (
MonitoringWindowClosed,
measure_monitoring_check,
@@ -893,7 +894,7 @@ def _arm_monitoring_checks(now: datetime, *, disabled_monitoring_organization_id
def run_due_report_checks(*, now: datetime | None = None, limit: int = MAX_CHECK_RUNS_PER_TICK) -> CheckRunSummary:
- """Expire stale checks and record due interim measurements during enabled Monitoring."""
+ """Record due final metric results and interim measurements during enabled Monitoring."""
now = now or timezone.now()
candidate_checks = SignalReportCheck.all_teams.filter(
@@ -915,7 +916,16 @@ def run_due_report_checks(*, now: datetime | None = None, limit: int = MAX_CHECK
"deferred": 0,
"cancelled": 0,
}
- for check in collect_due_checks(now, limit=limit, disabled_monitoring_organization_ids=disabled_organizations):
+ final = run_due_final_metrics(now=now, deadline=deadline, limit=limit)
+ counts.update(
+ passed=final.passed,
+ failed=final.failed,
+ inconclusive=final.inconclusive,
+ errored=final.errored,
+ deferred=final.deferred,
+ )
+ remaining = max(0, limit - sum(counts.values()))
+ for check in collect_due_checks(now, limit=remaining, disabled_monitoring_organization_ids=disabled_organizations):
if time.monotonic() >= deadline:
break
try:
diff --git a/products/signals/backend/report_check_final_metrics.py b/products/signals/backend/report_check_final_metrics.py
new file mode 100644
index 000000000000..9a9a00112b43
--- /dev/null
+++ b/products/signals/backend/report_check_final_metrics.py
@@ -0,0 +1,222 @@
+from __future__ import annotations
+
+import math
+import time
+from dataclasses import replace
+from datetime import datetime, timedelta
+from functools import partial
+from typing import cast
+from uuid import UUID, uuid5
+
+from django.db import transaction
+from django.db.models import (
+ Case,
+ Count,
+ DateTimeField,
+ Exists,
+ ExpressionWrapper,
+ F,
+ Func,
+ OuterRef,
+ Subquery,
+ TextField,
+ Value,
+ When,
+ Window,
+)
+from django.db.models.functions import Cast, Coalesce, RowNumber
+from django.utils import timezone
+
+import structlog
+
+from posthog.dataclasses import frozen
+
+from products.signals.backend.models import SignalReport, SignalReportCheck, SignalReportMonitoringResult
+from products.signals.backend.report_check_metric_evaluation import FinalMetricCheck, evaluate_final_metric
+from products.signals.backend.report_check_telemetry import capture_report_monitoring_evaluated
+from products.signals.backend.report_content_gates import team_report_monitoring_enabled
+from products.signals.backend.report_lifecycle import check_monitoring_disabled_organizations
+from products.signals.backend.report_metric_refresh import MetricQueryBudgetExhausted
+from products.signals.backend.report_monitoring_attempts import check_definition_version, current_monitoring_attempt
+from products.signals.backend.report_monitoring_results import (
+ MONITORING_RESULT_ELIGIBLE_STATUSES,
+ InvalidMonitoringResult,
+ MonitoringResultConflict,
+ save_monitoring_result,
+)
+
+logger = structlog.get_logger(__name__)
+FINAL_METRIC_RETRY_DELAYS = (timedelta(minutes=5), timedelta(minutes=15))
+MAX_FINAL_METRICS_PER_TEAM = 10
+
+
+@frozen
+class DueFinalMetric:
+ member: FinalMetricCheck
+ submission_id: UUID
+
+
+@frozen
+class FinalMetricSummary:
+ passed: int = 0
+ failed: int = 0
+ inconclusive: int = 0
+ errored: int = 0
+ deferred: int = 0
+
+
+def collect_due_final_metrics(now: datetime, *, limit: int, deadline: float = math.inf) -> list[DueFinalMetric]:
+ if limit <= 0:
+ return []
+ candidates = (
+ SignalReportCheck.all_teams.filter(
+ kind=SignalReportCheck.Kind.METRIC_THRESHOLD,
+ status__in=MONITORING_RESULT_ELIGIBLE_STATUSES,
+ soak_minutes__isnull=False,
+ report__status=SignalReport.Status.MONITORING,
+ report__monitoring_attempts__team_id=F("team_id"),
+ report__monitoring_attempts__monitoring_started_at=F("report__monitoring_started_at"),
+ )
+ .annotate(
+ _attempt_id=F("report__monitoring_attempts__id"),
+ _member_version=Func(
+ F("report__monitoring_attempts__check_versions"),
+ Cast(F("id"), output_field=TextField()),
+ function="jsonb_extract_path_text",
+ output_field=TextField(),
+ ),
+ _window_end=ExpressionWrapper(
+ F("report__monitoring_started_at") + F("soak_minutes") * Value(timedelta(minutes=1)),
+ output_field=DateTimeField(),
+ ),
+ )
+ .filter(_member_version__isnull=False, _window_end__lte=now)
+ )
+ results = SignalReportMonitoringResult.objects.unscoped().filter(
+ team_id=OuterRef("team_id"), attempt_id=OuterRef("_attempt_id"), report_check_id=OuterRef("id")
+ )
+ errors = results.filter(evidence_status="errored")
+ disabled = check_monitoring_disabled_organizations(candidates)
+ candidates = (
+ candidates.exclude(report__team__organization_id__in=disabled)
+ .annotate(
+ _settled=Exists(results.exclude(evidence_status="errored")),
+ _error_count=Coalesce(
+ Subquery(errors.order_by().values("report_check_id").annotate(total=Count("id")).values("total")[:1]),
+ Value(0),
+ ),
+ _last_error_at=Subquery(errors.order_by("-created_at", "-id").values("evaluated_at")[:1]),
+ )
+ .filter(_settled=False, _error_count__lte=len(FINAL_METRIC_RETRY_DELAYS))
+ .annotate(
+ _retry_at=Case(
+ *[
+ When(_error_count=index + 1, then=F("_last_error_at") + Value(delay))
+ for index, delay in enumerate(FINAL_METRIC_RETRY_DELAYS)
+ ],
+ default=Value(now),
+ output_field=DateTimeField(),
+ )
+ )
+ .filter(_retry_at__lte=now)
+ .annotate(
+ _team_rank=Window(
+ expression=RowNumber(),
+ partition_by=[F("team_id")],
+ order_by=[F("_window_end").asc(), F("id").asc()],
+ )
+ )
+ .select_related("report", "report__team")
+ .order_by("_team_rank", "_window_end", "id")
+ )
+ due: list[DueFinalMetric] = []
+ per_team: dict[int, int] = {}
+ for check in candidates.iterator(chunk_size=limit * MAX_FINAL_METRICS_PER_TEAM):
+ if time.monotonic() >= deadline:
+ break
+ if per_team.get(check.team_id, 0) >= MAX_FINAL_METRICS_PER_TEAM:
+ continue
+ version = cast(str, check._member_version)
+ if check_definition_version(check) != version:
+ continue
+ attempt_id = cast(UUID, check._attempt_id)
+ start = check.report.monitoring_started_at
+ assert start is not None
+ due.append(
+ DueFinalMetric(
+ member=FinalMetricCheck(
+ check=check,
+ attempt_id=attempt_id,
+ definition_version=version,
+ window_start=start,
+ window_end=cast(datetime, check._window_end),
+ ),
+ # Competing ticks share an ID so only one result consumes this logical retry.
+ submission_id=uuid5(attempt_id, f"metric:{check.id}:{version}:{check._error_count}"),
+ )
+ )
+ per_team[check.team_id] = per_team.get(check.team_id, 0) + 1
+ if len(due) >= limit:
+ break
+ return due
+
+
+def _prepare_final_metric(due: DueFinalMetric) -> bool:
+ member = due.member
+ with transaction.atomic():
+ report = SignalReport.objects.select_for_update(no_key=True).get(
+ id=member.check.report_id, team_id=member.check.team_id
+ )
+ if (
+ report.status != SignalReport.Status.MONITORING
+ or report.monitoring_started_at != member.window_start
+ or not team_report_monitoring_enabled(report.team_id)
+ ):
+ return False
+ attempt = current_monitoring_attempt(report)
+ if attempt is None or attempt.id != member.attempt_id:
+ return False
+ results = SignalReportMonitoringResult.objects.for_team(report.team_id).filter(
+ report_id=report.id, attempt_id=attempt.id, report_check_id=member.check.id
+ )
+ if (
+ results.filter(submission_id=due.submission_id).exists()
+ or results.exclude(evidence_status="errored").exists()
+ ):
+ return False
+ check = (
+ SignalReportCheck.objects.for_team(report.team_id)
+ .select_for_update()
+ .get(id=member.check.id, report_id=report.id)
+ )
+ if check_definition_version(check) != member.definition_version:
+ return False
+ return check.status in MONITORING_RESULT_ELIGIBLE_STATUSES
+
+
+def run_due_final_metrics(*, now: datetime, deadline: float, limit: int) -> FinalMetricSummary:
+ counts = {"passed": 0, "failed": 0, "inconclusive": 0, "errored": 0, "deferred": 0}
+ for due in collect_due_final_metrics(now, limit=limit, deadline=deadline):
+ if time.monotonic() >= deadline:
+ break
+ try:
+ if not _prepare_final_metric(due):
+ counts["deferred"] += 1
+ continue
+ evaluation = evaluate_final_metric(
+ due.member, submission_id=due.submission_id, evaluated_at=timezone.now(), deadline=deadline
+ )
+ result = save_monitoring_result(replace(evaluation, evaluated_at=timezone.now()))
+ transaction.on_commit(
+ partial(capture_report_monitoring_evaluated, due.member.check.report.team, due.member.check, result)
+ )
+ counts[result.outcome] += 1
+ except (MetricQueryBudgetExhausted, InvalidMonitoringResult, MonitoringResultConflict):
+ counts["deferred"] += 1
+ except Exception:
+ logger.exception(
+ "signals.monitoring.final_metric_step_failed",
+ team_id=due.member.check.team_id,
+ check_id=str(due.member.check.id),
+ )
+ return FinalMetricSummary(**counts)
diff --git a/products/signals/backend/report_check_metric_evaluation.py b/products/signals/backend/report_check_metric_evaluation.py
new file mode 100644
index 000000000000..21c9540c98f7
--- /dev/null
+++ b/products/signals/backend/report_check_metric_evaluation.py
@@ -0,0 +1,186 @@
+from __future__ import annotations
+
+import json
+import hashlib
+from dataclasses import replace
+from datetime import datetime
+from typing import TYPE_CHECKING
+from uuid import UUID
+
+import structlog
+
+from posthog.clickhouse.client.connection import ClickHouseUser
+from posthog.clickhouse.workload import Workload
+from posthog.dataclasses import frozen
+
+from products.signals.backend.report_check_evaluation import evaluate_check_value
+from products.signals.backend.report_check_evidence import (
+ EvidenceScope,
+ EvidenceSource,
+ MetricEvidence,
+ qualify_metric_evidence,
+)
+from products.signals.backend.report_check_timing import monitoring_metric_query
+from products.signals.backend.report_checks import (
+ MetricThresholdConfig,
+ monitoring_comparison,
+ parse_check_config,
+ validate_metric_check_for_write,
+)
+from products.signals.backend.report_metric_refresh import MetricQueryBudgetExhausted, measure_metric
+from products.signals.backend.report_monitoring_results import MonitoringEvaluation
+
+if TYPE_CHECKING:
+ from products.signals.backend.models import SignalReportCheck
+
+logger = structlog.get_logger(__name__)
+
+
+@frozen
+class FinalMetricCheck:
+ check: SignalReportCheck
+ attempt_id: UUID
+ definition_version: str
+ window_start: datetime
+ window_end: datetime
+
+
+def metric_evidence_source(config: MetricThresholdConfig) -> EvidenceSource:
+ assert config.query is not None
+ source = config.query["source"]
+ series = source["series"]
+ if len(series) != 1 or source.get("samplingFactor") not in (None, 1):
+ return "unsupported"
+ if any((source.get("trendsFilter") or {}).get(key) for key in ("formula", "formulas", "formulaNodes")):
+ return "unsupported"
+ entity = series[0]
+ if entity.get("math_property") == "$session_duration" or entity.get("math_property_type") == "session_properties":
+ return "unsupported"
+ if entity.get("math") in (None, "total"):
+ return "count"
+ if (
+ entity.get("math_property_type") not in (None, "event_properties")
+ or not isinstance(entity.get("math_property"), str)
+ or not entity["math_property"].strip()
+ ):
+ return "unsupported"
+ if entity.get("math") == "sum":
+ return "sum"
+ if entity.get("math") == "avg":
+ return "average"
+ return "unsupported"
+
+
+def evaluate_final_metric(
+ member: FinalMetricCheck, *, submission_id: UUID, evaluated_at: datetime, deadline: float
+) -> MonitoringEvaluation:
+ check = member.check
+ evaluation = MonitoringEvaluation(
+ team_id=check.team_id,
+ report_id=check.report_id,
+ attempt_id=member.attempt_id,
+ check_id=check.id,
+ submission_id=submission_id,
+ definition_version=member.definition_version,
+ window_start=member.window_start,
+ window_end=member.window_end,
+ evaluated_at=evaluated_at,
+ evidence_status="unsupported",
+ evidence_reason="invalid_configuration",
+ expectation_matches=None,
+ explanation="The check's metric configuration cannot be evaluated. Review its query and target.",
+ )
+ try:
+ config = parse_check_config(check.kind, check.config)
+ assert isinstance(config, MetricThresholdConfig)
+ validate_metric_check_for_write(config)
+ except ValueError:
+ return evaluation
+ if config.monitoring is None:
+ return replace(
+ evaluation,
+ evidence_status="unsupported",
+ evidence_reason="missing_monitoring_definition",
+ expectation_matches=None,
+ explanation="The check needs a Monitoring measurement definition before its target can be evaluated.",
+ )
+ source = metric_evidence_source(config)
+ if source == "unsupported" or source == "average":
+ return replace(
+ evaluation,
+ evidence_status="unsupported" if source == "unsupported" else "insufficient",
+ evidence_reason="unsupported_source" if source == "unsupported" else "minimum_denominator_unspecified",
+ expectation_matches=None,
+ explanation=(
+ "This metric needs an evidence adapter before its target can be evaluated."
+ if source == "unsupported"
+ else "This average needs a source-specific minimum denominator before it can establish success or failure."
+ ),
+ )
+ scope = EvidenceScope(
+ team_id=check.team_id,
+ population=f"check:{check.id}:{member.definition_version}",
+ start=member.window_start,
+ end=member.window_end,
+ )
+ if member.window_start == member.window_end:
+ return replace(
+ evaluation,
+ evidence_status="insufficient",
+ evidence_reason="empty_window",
+ expectation_matches=None,
+ explanation="The observation window is empty and cannot establish success or failure.",
+ )
+ assert config.query is not None
+ query = monitoring_metric_query(config.query, start=member.window_start, cutoff=member.window_end)
+ query_hash = hashlib.sha256(json.dumps(query, sort_keys=True, separators=(",", ":")).encode()).hexdigest()
+ references = (f"signal-report-check:{check.id}:{member.definition_version}", f"trends-query:sha256:{query_hash}")
+ try:
+ measurement = measure_metric(
+ query,
+ check.report.team,
+ deadline=deadline,
+ include_series=False,
+ require_full_timeout=True,
+ ch_user=ClickHouseUser.SIGNALS,
+ workload=Workload.OFFLINE,
+ )
+ if measurement.measured_at < member.window_end:
+ return replace(
+ evaluation,
+ evidence_status="insufficient",
+ evidence_reason="measurement_unavailable",
+ expectation_matches=None,
+ explanation="The query result predates the end of the observation window. A fresh result is needed.",
+ evidence_refs=references,
+ )
+ qualification = qualify_metric_evidence(MetricEvidence(scope=scope, source=source, value=measurement.value))
+ comparison = monitoring_comparison(config, member.window_end - member.window_start)
+ verdict = evaluate_check_value(comparison=comparison, observed_value=measurement.value, subject=check.title)
+ except MetricQueryBudgetExhausted:
+ raise
+ except Exception:
+ logger.exception("signals.monitoring.final_metric_query_failed", team_id=check.team_id, check_id=str(check.id))
+ return replace(
+ evaluation,
+ evidence_status="errored",
+ evidence_reason="query_error",
+ expectation_matches=None,
+ explanation="The final metric query could not be completed. Review the check's query.",
+ evidence_refs=references,
+ )
+ explanation = f"{verdict.explanation}. Query measured at {measurement.measured_at.isoformat()}."
+ if qualification.status == "insufficient":
+ explanation = f"The final value was {measurement.value}, measured at {measurement.measured_at.isoformat()}. Relevant activity for this metric's population has not been established; the value cannot establish success or failure."
+ elif qualification.status == "unsupported":
+ explanation = f"The final value was {measurement.value}, measured at {measurement.measured_at.isoformat()}. This value cannot establish success or failure under the metric's evidence policy."
+ return replace(
+ evaluation,
+ evidence_status=qualification.status,
+ evidence_reason=qualification.reason,
+ expectation_matches=verdict.outcome == "passed"
+ if qualification.status in {"sufficient", "insufficient"}
+ else None,
+ explanation=explanation,
+ evidence_refs=references,
+ )
diff --git a/products/signals/backend/report_check_telemetry.py b/products/signals/backend/report_check_telemetry.py
index e6354671ed9a..3082535a19ff 100644
--- a/products/signals/backend/report_check_telemetry.py
+++ b/products/signals/backend/report_check_telemetry.py
@@ -1,13 +1,14 @@
"""Adoption telemetry for the forward-looking checks attached to signal reports.
-Five events follow a check through its life. `signals_report_check_created` fires when an author
+Events follow a check through its life. `signals_report_check_created` fires when an author
writes one. `signals_report_check_dispatch` fires each time the coordinator tries to hand an `agent`
check to a scout run, whether the run started or the attempt was deferred. `signals_report_check_evaluated`
fires for each verdict a run records. `signals_report_checks_expired` fires when checks retire at
their horizon without a verdict. `signals_report_check_skipped` fires when research wrote a check
the project cannot run, so it was never stored. Together they say how many reports get a check, which kind their
authors reach for, how long the soak window really is, how often a claim stops holding, and where a
-check that never reported got stuck.
+check that never reported got stuck. `signals_report_monitoring_evaluated` records the final evidence
+outcome for each implementation-attempt submission.
All are best-effort: telemetry must never fail the write it describes, so every call swallows its
own errors.
@@ -23,12 +24,15 @@ import structlog
import posthoganalytics
from posthog.event_usage import groups
+from posthog.ph_client import ph_background_capture
from products.signals.backend.models import SignalReportCheck
if TYPE_CHECKING:
from posthog.models import Team
+ from products.signals.backend.models import SignalReportMonitoringResult
+
logger = structlog.get_logger(__name__)
@@ -56,6 +60,32 @@ def _identity(team: Team, check: SignalReportCheck) -> dict[str, object]:
}
+def capture_report_monitoring_evaluated(
+ team: Team, check: SignalReportCheck, result: SignalReportMonitoringResult
+) -> None:
+ try:
+ ph_background_capture()(
+ event="signals_report_monitoring_evaluated",
+ distinct_id=str(team.uuid),
+ uuid=str(result.id),
+ properties={
+ **_identity(team, check),
+ "attempt_id": str(result.attempt_id),
+ "submission_id": str(result.submission_id),
+ "outcome": result.outcome,
+ "evidence_status": result.evidence_status,
+ "evidence_reason": result.evidence_reason,
+ "window_minutes": (result.window_end - result.window_start).total_seconds() / 60,
+ "evaluation_delay_minutes": (result.evaluated_at - result.window_end).total_seconds() / 60,
+ },
+ groups=groups(team.organization, team),
+ )
+ except Exception:
+ logger.exception(
+ "signals.report_check.monitoring_capture_failed", check_id=str(check.id), team_id=check.team_id
+ )
+
+
def capture_report_check_created(team: Team, check: SignalReportCheck) -> None:
"""`signals_report_check_created`: an author scheduled a re-measurement of a report's claim.
diff --git a/products/signals/backend/report_metric_refresh.py b/products/signals/backend/report_metric_refresh.py
index fdd02c15616a..70a1fac203fe 100644
--- a/products/signals/backend/report_metric_refresh.py
+++ b/products/signals/backend/report_metric_refresh.py
@@ -112,6 +112,7 @@ def _run_metric_query(
display: ChartDisplayType,
*,
deadline: float,
+ require_full_timeout: bool = False,
ch_user: ClickHouseUser = ClickHouseUser.DEFAULT,
workload: Workload = Workload.DEFAULT,
) -> tuple[dict, datetime | None]:
@@ -141,6 +142,8 @@ def _run_metric_query(
if math.isinf(remaining)
else min(REPORT_METRIC_REFRESH_QUERY_TIMEOUT_SECONDS, int(remaining / source_count))
)
+ if require_full_timeout and timeout < REPORT_METRIC_REFRESH_QUERY_TIMEOUT_SECONDS:
+ raise MetricQueryBudgetExhausted("report metric query needs its full timeout allowance")
tag_queries(trigger="signals_report_metric_refresh")
response = run_cached_trends_query(
query=shaped_source,
@@ -170,11 +173,18 @@ def whole_window_value(
team: Team,
*,
deadline: float = math.inf,
+ require_full_timeout: bool = False,
ch_user: ClickHouseUser = ClickHouseUser.DEFAULT,
workload: Workload = Workload.DEFAULT,
) -> tuple[float, datetime]:
first_series, last_refresh = _run_metric_query(
- query, team, ChartDisplayType.BOLD_NUMBER, deadline=deadline, ch_user=ch_user, workload=workload
+ query,
+ team,
+ ChartDisplayType.BOLD_NUMBER,
+ deadline=deadline,
+ require_full_timeout=require_full_timeout,
+ ch_user=ch_user,
+ workload=workload,
)
value = _finite_number(first_series.get("aggregated_value"), what="aggregate")
measured_at = last_refresh if isinstance(last_refresh, datetime) else timezone.now()
@@ -208,10 +218,18 @@ def measure_metric(
*,
deadline: float,
include_series: bool,
+ require_full_timeout: bool = False,
ch_user: ClickHouseUser = ClickHouseUser.DEFAULT,
workload: Workload = Workload.DEFAULT,
) -> MetricMeasurement:
- value, measured_at = whole_window_value(query, team, deadline=deadline, ch_user=ch_user, workload=workload)
+ value, measured_at = whole_window_value(
+ query,
+ team,
+ deadline=deadline,
+ require_full_timeout=require_full_timeout,
+ ch_user=ch_user,
+ workload=workload,
+ )
series: list[float] | None
if not include_series or time.monotonic() >= deadline:
return MetricMeasurement(value=value, measured_at=measured_at, series=None)
diff --git a/products/signals/backend/report_monitoring_results.py b/products/signals/backend/report_monitoring_results.py
index 98d254b9b839..eabd43f816d2 100644
--- a/products/signals/backend/report_monitoring_results.py
+++ b/products/signals/backend/report_monitoring_results.py
@@ -13,6 +13,12 @@ from products.signals.backend.report_check_evidence import EvidenceStatus
from products.signals.backend.report_content_gates import team_report_monitoring_enabled
from products.signals.backend.report_monitoring_attempts import check_definition_version, current_monitoring_attempt
+MONITORING_RESULT_ELIGIBLE_STATUSES = (
+ *SignalReportCheck.OPEN_STATUSES,
+ SignalReportCheck.Status.ERRORED,
+ SignalReportCheck.Status.EXPIRED,
+)
+
class InvalidMonitoringResult(ValueError):
pass
@@ -111,8 +117,8 @@ def save_monitoring_result(evaluation: MonitoringEvaluation) -> SignalReportMoni
now = timezone.now()
if not team_report_monitoring_enabled(evaluation.team_id):
raise InvalidMonitoringResult("Monitoring is disabled.")
- if check is None or check.status not in SignalReportCheck.OPEN_STATUSES or check.expires_at <= now:
- raise InvalidMonitoringResult("The check is no longer open and unexpired.")
+ if check is None or check.status not in MONITORING_RESULT_ELIGIBLE_STATUSES:
+ raise InvalidMonitoringResult("The check was cancelled or already completed.")
if check_definition_version(check) != evaluation.definition_version:
raise InvalidMonitoringResult("The authored definition changed.")
if check.soak_minutes is None:
diff --git a/products/signals/backend/test/test_report_check_concurrency.py b/products/signals/backend/test/test_report_check_concurrency.py
index 8483aa9e7114..d9d608246766 100644
--- a/products/signals/backend/test/test_report_check_concurrency.py
+++ b/products/signals/backend/test/test_report_check_concurrency.py
@@ -169,7 +169,7 @@ class TestReportCheckConcurrency(NonAtomicBaseTest):
explanation="The expectation holds.",
)
- @parameterized.expand([("cancelled",), ("definition",), ("expired",), ("flag_off",)])
+ @parameterized.expand([("cancelled",), ("definition",), ("completed",), ("flag_off",), ("errored",), ("expired",)])
def test_final_result_rechecks_check_invalidation_after_a_lock_wait(self, change: str) -> None:
with (
patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True) as flag,
@@ -187,13 +187,25 @@ class TestReportCheckConcurrency(NonAtomicBaseTest):
)
elif change == "definition":
checks.update(title="A different expectation")
- elif change == "expired":
- checks.update(expires_at=timezone.now())
+ elif change in {"errored", "expired", "completed"}:
+ checks.update(
+ status=SignalReportCheck.Status.PASSED if change == "completed" else change,
+ expires_at=timezone.now(),
+ )
else:
flag.return_value = False
- with self.assertRaises(InvalidMonitoringResult):
- future.result(timeout=10)
- assert not SignalReportMonitoringResult.objects.for_team(self.team.id).exists()
+ if change in {"errored", "expired"}:
+ result = future.result(timeout=10)
+ assert isinstance(result, SignalReportMonitoringResult)
+ assert result.outcome == "passed"
+ self.check.refresh_from_db()
+ assert self.check.status == change
+ else:
+ with self.assertRaises(InvalidMonitoringResult):
+ future.result(timeout=10)
+ assert SignalReportMonitoringResult.objects.for_team(self.team.id).exists() == (
+ change in {"errored", "expired"}
+ )
@parameterized.expand([("resolved",), ("suppressed",), ("reopened",)])
def test_final_result_rechecks_report_state_after_a_lock_wait(self, change: str) -> None:
diff --git a/products/signals/backend/test/test_report_check_final_metrics.py b/products/signals/backend/test/test_report_check_final_metrics.py
new file mode 100644
index 000000000000..e05bfbf70083
--- /dev/null
+++ b/products/signals/backend/test/test_report_check_final_metrics.py
@@ -0,0 +1,438 @@
+import time
+from datetime import timedelta
+from uuid import uuid4
+
+import time_machine
+from posthog.test.base import BaseTest, _create_event
+from unittest.mock import patch
+
+from django.test import SimpleTestCase
+from django.utils import timezone
+
+from parameterized import parameterized
+
+from posthog.clickhouse.client.connection import ClickHouseUser
+from posthog.clickhouse.workload import Workload
+from posthog.models import Organization, Team
+
+from products.signals.backend.models import (
+ SignalReport,
+ SignalReportCheck,
+ SignalReportMonitoringAttempt,
+ SignalReportMonitoringResult,
+)
+from products.signals.backend.report_check_execution import run_due_report_checks
+from products.signals.backend.report_check_final_metrics import collect_due_final_metrics, run_due_final_metrics
+from products.signals.backend.report_check_metric_evaluation import FinalMetricCheck, metric_evidence_source
+from products.signals.backend.report_checks import MAX_CONSECUTIVE_CHECK_ERRORS, MetricThresholdConfig
+from products.signals.backend.report_metric_refresh import MetricMeasurement, MetricQueryBudgetExhausted
+from products.signals.backend.report_monitoring_attempts import check_definition_version, complete_implementation
+from products.signals.backend.test.report_metric_test_fixtures import trends_metric_query
+
+_MEASURE = "products.signals.backend.report_check_metric_evaluation.measure_metric"
+_FLAG = "products.signals.backend.report_content_gates.feature_enabled_or_false"
+
+
+class TestMetricEvidenceAdapter(SimpleTestCase):
+ @parameterized.expand(
+ [
+ ("default_count", None, None, "count"),
+ ("raw_count", "total", None, "count"),
+ ("event_sum", "sum", "amount", "sum"),
+ ("average_needs_policy", "avg", "amount", "average"),
+ ("session_sum", "sum", "$session_duration", "unsupported"),
+ ("session_count", "total", "$session_duration", "unsupported"),
+ ("implicit_session_count", None, "$session_duration", "unsupported"),
+ ("typed_session_count", "total", "$pageview_count", "unsupported", "session_properties"),
+ ("event_property_count", "total", "$pageview_count", "count", "event_properties"),
+ ("missing_property", "sum", None, "unsupported"),
+ ("unique_people", "dau", None, "unsupported"),
+ ("custom_expression", "hogql", None, "unsupported"),
+ ]
+ )
+ def test_source_semantics_do_not_follow_display_format(
+ self,
+ _name: str,
+ math: str | None,
+ property_name: str | None,
+ expected: str,
+ property_type: str | None = None,
+ ) -> None:
+ entity = {"kind": "EventsNode", "event": "checkout", "math": math}
+ if property_name is not None:
+ entity["math_property"] = property_name
+ if property_type is not None:
+ entity["math_property_type"] = property_type
+ config = MetricThresholdConfig.model_validate(
+ {"query": trends_metric_query(series=[entity]), "comparison": {"operator": "gte", "value": 1}}
+ )
+ assert metric_evidence_source(config) == expected
+
+ @parameterized.expand([("sampled",), ("formula",), ("person_property",)])
+ def test_unsupported_shapes_cannot_qualify_through_a_scalar(self, shape: str) -> None:
+ query = trends_metric_query(
+ series=[{"kind": "EventsNode", "event": "checkout", "math": "sum", "math_property": "amount"}]
+ )
+ if shape == "sampled":
+ query["source"]["samplingFactor"] = 0.5
+ elif shape == "formula":
+ query["source"]["trendsFilter"] = {"formula": "A * 2"}
+ else:
+ query["source"]["series"][0]["math_property_type"] = "person_properties"
+ config = MetricThresholdConfig.model_validate({"query": query, "comparison": {"operator": "gte", "value": 1}})
+ assert metric_evidence_source(config) == "unsupported"
+
+
+class TestFinalMetricExecution(BaseTest):
+ def setUp(self) -> None:
+ super().setUp()
+ self.now = timezone.now()
+ self.enterContext(time_machine.travel(self.now, tick=False))
+ self.flag = self.enterContext(patch(_FLAG, return_value=True))
+ self.report = SignalReport.objects.create(
+ team=self.team, status=SignalReport.Status.READY, title="Checkout succeeds"
+ )
+
+ def _member(
+ self,
+ *,
+ soak_minutes: int | None = 60,
+ math: str = "total",
+ math_property: str = "amount",
+ math_property_type: str | None = None,
+ definition: bool = True,
+ formula: bool = False,
+ scaling: str = "fixed",
+ event: str = "checkout",
+ ) -> FinalMetricCheck:
+ config = {
+ "query": trends_metric_query(
+ series=[{"kind": "EventsNode", "event": event, "math": math, "math_property": math_property}]
+ ),
+ "comparison": {"operator": "gte", "value": 10},
+ }
+ if math_property_type is not None:
+ config["query"]["source"]["series"][0]["math_property_type"] = math_property_type
+ if definition:
+ config["monitoring"] = {"reference_period_minutes": 120, "threshold_scaling": scaling}
+ if formula:
+ config["query"]["source"]["trendsFilter"] = {"formula": "A * 2"}
+ check = SignalReportCheck.objects.for_team(self.team.id).create(
+ team_id=self.team.id,
+ report=self.report,
+ title="Successful checkouts",
+ kind="metric_threshold",
+ config=config,
+ soak_minutes=soak_minutes,
+ next_run_at=self.now,
+ expires_at=self.now + timedelta(days=30),
+ runs_remaining=3,
+ )
+ start = self.now - timedelta(hours=2)
+ attempt = complete_implementation(team_id=self.team.id, report_id=self.report.id, monitoring_started_at=start)
+ assert attempt is not None
+ check.refresh_from_db()
+ check.report.refresh_from_db()
+ return FinalMetricCheck(
+ check=check,
+ attempt_id=attempt.id,
+ definition_version=attempt.check_versions[str(check.id)],
+ window_start=start,
+ window_end=start + timedelta(minutes=soak_minutes or 0),
+ )
+
+ def _run(self) -> None:
+ run_due_final_metrics(now=timezone.now(), deadline=time.monotonic() + 30, limit=50)
+
+ @parameterized.expand(
+ [
+ ("pass", "total", 12, "passed"),
+ ("failure", "total", 3, "failed"),
+ ("sum", "sum", 12.5, "passed"),
+ ("zero_count", "total", 0, "inconclusive"),
+ ("zero_sum", "sum", 0, "inconclusive"),
+ ("negative_sum", "sum", -1, "inconclusive"),
+ ]
+ )
+ def test_final_results_separate_evidence_from_matching_and_do_not_complete_reports(
+ self, _name: str, math: str, value: float, outcome: str
+ ) -> None:
+ member = self._member(math=math)
+ with (
+ patch("products.signals.backend.report_check_telemetry.ph_background_capture") as capture,
+ self.captureOnCommitCallbacks(execute=True),
+ patch(_MEASURE, return_value=MetricMeasurement(value=value, measured_at=self.now, series=None)) as measure,
+ ):
+ summary = run_due_report_checks(now=self.now)
+ self._run()
+ result = SignalReportMonitoringResult.objects.for_team(self.team.id).get()
+ assert result.outcome == outcome
+ event = capture.return_value.call_args.kwargs
+ assert event["event"] == "signals_report_monitoring_evaluated"
+ assert event["uuid"] == str(result.id)
+ assert event["properties"]["outcome"] == outcome
+ assert result.window_start == member.window_start and result.window_end == member.window_end
+ assert result.definition_version == member.definition_version
+ assert str(value) in result.explanation
+ assert len(result.evidence_refs) == 2
+ query = measure.call_args.args[0]
+ assert query["source"]["dateRange"] == {
+ "date_from": member.window_start.isoformat(),
+ "date_to": member.window_end.isoformat(),
+ "explicitDate": True,
+ "excludeIncompletePeriods": False,
+ }
+ assert measure.call_args.kwargs["workload"] == Workload.OFFLINE
+ assert measure.call_args.kwargs["ch_user"] == ClickHouseUser.SIGNALS
+ assert measure.call_count == 1
+ assert sum([summary.passed, summary.failed, summary.inconclusive]) == 1
+ self.report.refresh_from_db()
+ member.check.refresh_from_db()
+ assert self.report.status == SignalReport.Status.MONITORING
+ assert member.check.runs_remaining == 3 and member.check.last_outcome is None
+ if value == 0:
+ assert result.evidence_status == "insufficient" and result.evidence_reason == "activity_unavailable"
+ elif value < 0:
+ assert result.evidence_status == "unsupported"
+
+ @parameterized.expand(
+ [
+ ("empty_window",),
+ ("missing_definition",),
+ ("formula",),
+ ("average",),
+ ("session_duration_count",),
+ ("session_property_count",),
+ ]
+ )
+ def test_unmeasurable_members_get_explicit_results_without_querying(self, variant: str) -> None:
+ self._member(
+ soak_minutes=0 if variant == "empty_window" else 60,
+ definition=variant != "missing_definition",
+ math="avg" if variant == "average" else "total",
+ math_property={
+ "session_duration_count": "$session_duration",
+ "session_property_count": "$pageview_count",
+ }.get(variant, "amount"),
+ math_property_type="session_properties" if variant == "session_property_count" else None,
+ formula=variant == "formula",
+ )
+ with patch(_MEASURE) as measure:
+ self._run()
+ self._run()
+ result = SignalReportMonitoringResult.objects.for_team(self.team.id).get()
+ assert result.outcome == "inconclusive"
+ assert (
+ result.evidence_reason
+ == {
+ "empty_window": "empty_window",
+ "missing_definition": "missing_monitoring_definition",
+ "formula": "unsupported_source",
+ "average": "minimum_denominator_unspecified",
+ "session_duration_count": "unsupported_source",
+ "session_property_count": "unsupported_source",
+ }[variant]
+ )
+ measure.assert_not_called()
+
+ @parameterized.expand([("exhausted", False), ("recovered", True)])
+ def test_query_errors_retry_three_times_on_fixed_windows_and_remain_durable(
+ self, _name: str, recovered: bool
+ ) -> None:
+ member = self._member()
+ responses = [
+ RuntimeError("private query detail"),
+ RuntimeError("private query detail"),
+ MetricMeasurement(value=12, measured_at=self.now, series=None)
+ if recovered
+ else RuntimeError("private query detail"),
+ ]
+ with patch(_MEASURE, side_effect=responses) as measure:
+ for minutes, expected_calls in [(0, 1), (4, 1), (5, 2), (19, 2), (20, 3), (120, 3)]:
+ with time_machine.travel(self.now + timedelta(minutes=minutes), tick=False):
+ self._run()
+ assert measure.call_count == expected_calls
+ results = list(SignalReportMonitoringResult.objects.for_team(self.team.id).order_by("created_at"))
+ assert len(results) == 3 and len({result.submission_id for result in results}) == 3
+ assert all(
+ result.window_start == member.window_start and result.window_end == member.window_end for result in results
+ )
+ assert [result.outcome for result in results] == ["errored", "errored", "passed" if recovered else "errored"]
+ assert all("private query detail" not in result.explanation for result in results)
+
+ @parameterized.expand([("fixed", "failed"), ("linear", "passed")])
+ def test_final_comparison_uses_the_authored_scaling_policy(self, scaling: str, outcome: str) -> None:
+ self._member(scaling=scaling)
+ with patch(_MEASURE, return_value=MetricMeasurement(value=6, measured_at=self.now, series=None)):
+ self._run()
+ assert SignalReportMonitoringResult.objects.for_team(self.team.id).get().outcome == outcome
+
+ def test_cached_predeadline_results_cannot_establish_a_final_pass(self) -> None:
+ member = self._member()
+ with patch(
+ _MEASURE,
+ return_value=MetricMeasurement(value=12, measured_at=member.window_end - timedelta(seconds=1), series=None),
+ ):
+ self._run()
+ result = SignalReportMonitoringResult.objects.for_team(self.team.id).get()
+ assert result.evidence_status == "insufficient" and result.expectation_matches is None
+
+ @parameterized.expand([("total", 3), ("sum", 6)])
+ def test_real_query_excludes_events_outside_the_fixed_window_and_population(self, math: str, value: int) -> None:
+ population = f"checkout-{uuid4()}"
+ member = self._member(math=math, event=population)
+ for timestamp, event, count in [
+ (member.window_start - timedelta(seconds=1), population, 20),
+ (member.window_start + timedelta(minutes=1), population, 3),
+ (member.window_end + timedelta(seconds=1), population, 20),
+ (member.window_start + timedelta(minutes=1), "unrelated", 20),
+ ]:
+ for _ in range(count):
+ _create_event(
+ team=self.team, event=event, distinct_id="test-user", timestamp=timestamp, properties={"amount": 2}
+ )
+ self._run()
+ result = SignalReportMonitoringResult.objects.for_team(self.team.id).get()
+ assert result.outcome == "failed" and result.evidence_status == "sufficient", result.explanation
+ assert str(float(value)) in result.explanation
+
+ def test_budget_deferral_does_not_spend_a_technical_retry(self) -> None:
+ self._member()
+ with patch(_MEASURE, side_effect=MetricQueryBudgetExhausted):
+ self._run()
+ assert not SignalReportMonitoringResult.objects.for_team(self.team.id).exists()
+ assert len(collect_due_final_metrics(self.now, limit=10)) == 1
+
+ @parameterized.expand([("short", 19.9, False), ("full", 20.0, True)])
+ def test_only_queries_with_the_full_timeout_can_spend_a_technical_retry(
+ self, _name: str, remaining_seconds: float, spends_retry: bool
+ ) -> None:
+ self._member()
+ with (
+ patch("time.monotonic", return_value=0),
+ patch(
+ "products.signals.backend.report_metric_refresh.run_cached_trends_query", side_effect=TimeoutError
+ ) as query,
+ ):
+ for _ in range(3):
+ summary = run_due_final_metrics(now=self.now, deadline=remaining_seconds, limit=50)
+ assert query.call_count == int(spends_retry)
+ results = SignalReportMonitoringResult.objects.for_team(self.team.id)
+ if spends_retry:
+ assert results.get().outcome == "errored"
+ assert query.call_args.kwargs["max_execution_time_seconds"] == 20
+ else:
+ assert summary.deferred == 1
+ assert not results.exists()
+ assert len(collect_due_final_metrics(self.now, limit=10)) == 1
+
+ @parameterized.expand([("cancelled",), ("archived",), ("definition",), ("disabled",), ("reopened",)])
+ def test_query_results_cannot_cross_lifecycle_or_definition_changes(self, change: str) -> None:
+ member = self._member()
+
+ def change_during_query(*args: object, **kwargs: object) -> MetricMeasurement:
+ if change == "cancelled":
+ SignalReportCheck.objects.for_team(self.team.id).filter(id=member.check.id).update(status="cancelled")
+ elif change == "definition":
+ SignalReportCheck.objects.for_team(self.team.id).filter(id=member.check.id).update(
+ title="A new expectation"
+ )
+ elif change == "disabled":
+ self.flag.return_value = False
+ else:
+ SignalReport.objects.filter(id=self.report.id, team_id=self.team.id).update(
+ status="suppressed" if change == "archived" else "ready"
+ )
+ return MetricMeasurement(value=12, measured_at=self.now, series=None)
+
+ with patch(_MEASURE, side_effect=change_during_query):
+ self._run()
+ assert not SignalReportMonitoringResult.objects.for_team(self.team.id).exists()
+
+ @parameterized.expand([("errored",), ("expired",)])
+ def test_final_verification_keeps_errored_and_expired_snapshot_members(self, status: str) -> None:
+ member = self._member()
+ SignalReportCheck.objects.for_team(self.team.id).filter(id=member.check.id).update(
+ status=status,
+ measurement_start_at=member.window_start,
+ consecutive_errors=MAX_CONSECUTIVE_CHECK_ERRORS,
+ expires_at=self.now - timedelta(minutes=1),
+ )
+ with patch(_MEASURE, return_value=MetricMeasurement(value=12, measured_at=self.now, series=None)):
+ self._run()
+ assert SignalReportMonitoringResult.objects.for_team(self.team.id).get().outcome == "passed"
+ member.check.refresh_from_db()
+ assert member.check.status == status
+ assert member.check.consecutive_errors == MAX_CONSECUTIVE_CHECK_ERRORS
+
+ def test_stale_collections_cannot_duplicate_a_logical_submission(self) -> None:
+ self._member()
+ due = collect_due_final_metrics(self.now, limit=10)
+ assert due[0].submission_id == collect_due_final_metrics(self.now, limit=10)[0].submission_id
+ with (
+ patch("products.signals.backend.report_check_final_metrics.collect_due_final_metrics", return_value=due),
+ patch(_MEASURE, side_effect=RuntimeError) as measure,
+ ):
+ self._run()
+ self._run()
+ assert measure.call_count == 1
+ assert SignalReportMonitoringResult.objects.for_team(self.team.id).count() == 1
+
+ @parameterized.expand([("disabled",), ("stale_definition",)])
+ def test_ineligible_members_cannot_fill_the_bounded_candidate_batch(self, variant: str) -> None:
+ member = self._member()
+ disabled_org = Organization.objects.create(name="Monitoring disabled")
+ start = self.now - timedelta(hours=3)
+ for _ in range(10):
+ team = Team.objects.create(organization=disabled_org)
+ report = SignalReport.objects.create(team=team, status="monitoring", monitoring_started_at=start)
+ check = SignalReportCheck.objects.for_team(team.id).create(
+ team_id=team.id,
+ report=report,
+ title="Old check",
+ kind="metric_threshold",
+ config=member.check.config,
+ soak_minutes=60,
+ next_run_at=self.now,
+ expires_at=self.now + timedelta(days=1),
+ )
+ SignalReportMonitoringAttempt.objects.for_team(team.id).create(
+ team_id=team.id,
+ report=report,
+ monitoring_started_at=start,
+ check_versions={str(check.id): check_definition_version(check)},
+ )
+ if variant == "stale_definition":
+ SignalReportCheck.objects.for_team(team.id).filter(id=check.id).update(title="Changed target")
+ with patch(
+ "products.signals.backend.report_lifecycle.organization_report_monitoring_enabled",
+ side_effect=lambda organization_id: variant != "disabled" or organization_id != disabled_org.id,
+ ):
+ due = collect_due_final_metrics(self.now, limit=1)
+ assert [entry.member.check.id for entry in due] == [member.check.id]
+
+ @parameterized.expand([("before_deadline",), ("missing_soak",), ("not_in_snapshot",), ("edited",), ("disabled",)])
+ def test_due_work_requires_current_snapshotted_obligations(self, variant: str) -> None:
+ member = self._member(
+ soak_minutes=180 if variant == "before_deadline" else None if variant == "missing_soak" else 60
+ )
+ if variant == "not_in_snapshot":
+ SignalReportCheck.objects.for_team(self.team.id).create(
+ team_id=self.team.id,
+ report=self.report,
+ title="Added after implementation",
+ kind="metric_threshold",
+ config=member.check.config,
+ soak_minutes=60,
+ next_run_at=self.now,
+ expires_at=self.now + timedelta(days=30),
+ )
+ SignalReportCheck.objects.for_team(self.team.id).filter(id=member.check.id).update(status="cancelled")
+ elif variant == "edited":
+ SignalReportCheck.objects.for_team(self.team.id).filter(id=member.check.id).update(title="Changed target")
+ elif variant == "disabled":
+ self.flag.return_value = False
+ with patch(_MEASURE) as measure:
+ self._run()
+ measure.assert_not_called()
+ assert not SignalReportMonitoringResult.objects.for_team(self.team.id).exists()
diff --git a/products/signals/backend/test/test_report_monitoring_attempts.py b/products/signals/backend/test/test_report_monitoring_attempts.py
index 22a076a70dc2..26ee940dfe98 100644
--- a/products/signals/backend/test/test_report_monitoring_attempts.py
+++ b/products/signals/backend/test/test_report_monitoring_attempts.py
@@ -155,10 +155,28 @@ class TestReportMonitoringAttempts(BaseTest):
evidence_refs=("observation:example",),
)
- @parameterized.expand([("metric_threshold",), ("agent",)])
- def test_final_result_is_durable_idempotent_and_does_not_complete_or_spend_runs(self, kind: str) -> None:
+ @parameterized.expand(
+ [
+ (kind, status, expired)
+ for kind in (SignalReportCheck.Kind.METRIC_THRESHOLD, SignalReportCheck.Kind.AGENT)
+ for status, expired in (
+ (SignalReportCheck.Status.PENDING, False),
+ (SignalReportCheck.Status.PENDING, True),
+ (SignalReportCheck.Status.ERRORED, False),
+ (SignalReportCheck.Status.ERRORED, True),
+ (SignalReportCheck.Status.EXPIRED, True),
+ )
+ ]
+ )
+ def test_final_result_is_durable_idempotent_and_does_not_complete_or_spend_runs(
+ self, kind: str, status: str, expired: bool
+ ) -> None:
check = self._check(kind=kind)
evaluation = self._evaluation(check)
+ check.status = status
+ if expired:
+ check.expires_at = self.now
+ check.save()
result = save_monitoring_result(evaluation)
assert save_monitoring_result(evaluation).id == result.id
@@ -181,7 +199,7 @@ class TestReportMonitoringAttempts(BaseTest):
self.report.refresh_from_db()
check.refresh_from_db()
assert self.report.status == SignalReport.Status.MONITORING
- assert check.status == SignalReportCheck.Status.PENDING
+ assert check.status == status
assert check.runs_remaining == 1
assert check.last_outcome is None
@@ -208,7 +226,7 @@ class TestReportMonitoringAttempts(BaseTest):
assert save_monitoring_result(evaluation).id == result.id
with self.assertRaises(MonitoringResultConflict):
save_monitoring_result(replace(evaluation, explanation="A different evaluation"))
- with self.assertRaises(InvalidMonitoringResult):
+ with self.assertRaises(MonitoringResultConflict if change == "expired" else InvalidMonitoringResult):
save_monitoring_result(replace(evaluation, submission_id=uuid4()))
assert SignalReportMonitoringResult.objects.for_team(self.team.id).count() == 1
@@ -251,8 +269,9 @@ class TestReportMonitoringAttempts(BaseTest):
("reopened",),
("new_attempt",),
("cancelled",),
- ("expired",),
- ("terminal",),
+ ("passed",),
+ ("failed",),
+ ("inconclusive",),
("definition",),
("snapshot",),
("nonmember",),
@@ -278,13 +297,9 @@ class TestReportMonitoringAttempts(BaseTest):
self.report.save(update_fields=self.report.transition_to(target))
if change == "new_attempt":
complete_implementation(team_id=self.team.id, report_id=self.report.id, monitoring_started_at=self.now)
- elif change in {"cancelled", "expired", "terminal", "definition"}:
- if change == "cancelled":
- check.status = SignalReportCheck.Status.CANCELLED
- elif change == "expired":
- check.expires_at = self.now
- elif change == "terminal":
- check.status = SignalReportCheck.Status.ERRORED
+ elif change in {"cancelled", "passed", "failed", "inconclusive", "definition"}:
+ if change != "definition":
+ check.status = change
else:
check.config = {"instructions": "A changed expectation"}
check.save()