:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(signals): record interim monitoring metric observations (#118718)
Mikayla Thompson7cb3b5b7c33c
diff --git a/docs/internal/signals-pr-lifecycle.md b/docs/internal/signals-pr-lifecycle.mdindex 6f8daa7c5cde..93155d6b22c8 100644--- a/docs/internal/signals-pr-lifecycle.md+++ b/docs/internal/signals-pr-lifecycle.md@@ -57,11 +57,17 @@ Protected research runs require analytics access before a person can resume or w## Monitoring metric definitions-Research should make every effort to complete `config.monitoring` for each metric check using the research evidence. If required facts remain unknown, explain what is missing in the rationale instead of inventing values. The definition records the target's `reference_period_minutes`, an explicit `threshold_scaling` policy, and `baseline_period_minutes` when a baseline is supplied. These periods are independent of the soak, run interval, expiry, and any future Monitoring deadline. No existing check receives an inferred definition.+Research should make every effort to complete `config.monitoring` for each metric check using the research evidence. If required facts remain unknown, explain what is missing in the rationale instead of inventing values. The definition records the target's `reference_period_minutes`, an explicit `threshold_scaling` policy, and `baseline_period_minutes` when a baseline is supplied. These reference periods are independent of the soak, which defines the Monitoring window, and of the run interval and safety expiry. No existing check receives an inferred definition.-The definition describes a measurement from Monitoring start through the observation time. A `linear` target scales by elapsed minutes divided by reference minutes and requires a single additive event count or property sum without a formula. Session-level sums are excluded because a session can contribute to more than one chart bucket. A `fixed` target keeps its bound as the observation period grows, including for percentages, averages, distinct counts, or explicitly fixed custom targets. Display format alone never selects the policy. The baseline retains its own measured period and is not presented as an observation from Monitoring.+The definition describes a measurement from Monitoring start through the observation time. A `linear` target scales by elapsed minutes divided by reference minutes and requires a single additive event count or property sum without a formula. Session-level sums are excluded because a session can contribute to more than one chart bucket. A `fixed` target keeps its bound as the observation period grows, including for percentages, averages, distinct counts, or explicitly fixed custom targets. Rolling weekly and monthly active-user queries are refused because their lookback overrides the Monitoring bounds; use a distinct user count instead. Display format alone never selects the policy. The baseline retains its own measured period and is not presented as an observation from Monitoring.-Expected impact keeps the activity chart visible and collapses the target period, scaling policy, baseline, and minimum wait under measurement details. Dynamic chart boundaries refresh at minute intervals so repeated views can reuse cached results. Bucket granularity follows the full displayed range and stays within the 1,000-point budget. The chart preserves date-range filters and spans one reference period on each side of Monitoring start, or now before Monitoring starts, extending through now if necessary. Start and now markers separate the surrounding activity from future dates, whose values remain empty. Compatible charts show target reference lines: additive linear targets scale to a full chart bucket, and fixed rates or property averages/quantiles retain their level. Fixed totals, distinct counts, unknown aggregations, linear calendar intervals longer than a day, and linear day buckets with unequal daylight-saving durations receive no reference line. These references do not impose per-bucket pass/fail rules. This read-time chart does not evaluate the Monitoring target or save a check result. The stored bounded query and whole-window comparison retain their existing Resolved execution meaning. Monitoring execution, scaled comparisons, saved interim observations, and final outcomes remain separate work.+Expected impact keeps the activity chart visible and collapses the target period, scaling policy, baseline, and minimum wait under measurement details. Dynamic chart boundaries refresh at minute intervals so repeated views can reuse cached results. Bucket granularity follows the full displayed range and stays within the 1,000-point budget. The chart preserves date-range filters and spans one reference period on each side of Monitoring start, or now before Monitoring starts, extending through now if necessary. Start and now markers separate the surrounding activity from future dates, whose values remain empty. Compatible charts show target reference lines: additive linear targets scale to a full chart bucket, and fixed rates or property averages/quantiles retain their level. Fixed totals, distinct counts, unknown aggregations, linear calendar intervals longer than a day, and linear day buckets with unequal daylight-saving durations receive no reference line. These references do not impose per-bucket pass/fail rules. This read-time chart does not evaluate the Monitoring target or save a check result. The stored bounded query and whole-window comparison retain their existing Resolved execution meaning. Final Monitoring outcomes remain separate work.++During Monitoring, checks with an explicit Monitoring definition sample at their configured run interval, or every 24 hours when none is set. The first observation is due one interval after Monitoring starts. Interim observations run only before Monitoring start plus the check's `soak_minutes`. The full trailing query window does not delay them. Interim queries explicitly use the offline ClickHouse workload; interactive metric refreshes retain their default workload. The next due time is capped at that soak deadline, but this PR does not execute a final check or conclude a report there. A missing or zero soak, or a soak no longer than the sampling interval, produces no interim observations. Each query pins both bounds to Monitoring start and one captured observation cutoff, preserving the population filters. Pinned ranges include partial periods so their bounds match the scaled target. Linear comparisons scale both bounds by elapsed time divided by the reference period, without rounding; fixed comparisons stay unchanged.++Each completed attempt appends a protected `check_observation` log entry with its exact bounds, soak deadline, display format, unit, effective comparison, value, and match result, or a sanitized query error. The log labels it as an interim measurement, shows whether the goal is on track, and displays time remaining when the measurement was taken. It does not assign a terminal outcome. Observations do not spend `runs_remaining`, change the report status, or emit recurrence signals. The next sample is due one interval after the cutoff; missed intervals do not create catch-up runs. Restoring Monitoring schedules from the latest attempt in that period, preserving a failed attempt's error streak and six-hour retry across suppression. Re-enabling the flag restores the measurement origin, the next attempt from the latest run, and the expiry from the original Monitoring start. This repair precedes the expiry sweep so a fallback expiry cannot shorten or extend the Monitoring horizon. A check that cannot fit its source queries into the remaining coordinator budget stays due without an observation or an error-budget change. Query failures retain the existing six-hour retry and consecutive-error limit, and the 90-day check horizon still applies. Writes reject a changed check, Monitoring period, report state, or flag, and duplicate workers cannot save the same attempt twice. Expiry sweeps and schedule repairs retain their check-row concurrency guards after database lock waits, so they cannot overwrite a concurrent cancellation, expiry extension, or saved observation.++With Monitoring disabled, observations disappear from the artefact API and counts. The existing Resolved executor uses the stored full-window query, comparison, and minimum soak. Observation values require the same query permissions as terminal check results. The inbox follows all artefact pages within the current project before deriving task runs, reviewers, and activity so observations cannot displace older report state. Polling waits for an in-flight paginated load to finish. With the Monitoring flag enabled, scouts see metric checks with an explicit definition as scheduled or due inside their Monitoring window; agent checks and metric checks without a definition remain paused until resolution. The soak deadline bounds interim execution; no final verdict, reopening, or follow-up report is added here.With `signals-report-metrics` enabled, research gathers short-window baselines before verification and prefers metric-based follow-up checks. It aims for practical 70-80% confidence in improvement, without claiming a calculated confidence score. Observed volume, recurrence, ingestion delay, and deployment timing justify the shortest useful query and wait. Frequent or high-volume checks can use 4-8 hours, allowing for several hours of deployment delay before the fix reaches users. Around 24 hours or several days require the corresponding activity or rollout evidence. Agent checks remain available when interpretation or unsupported measurements prevent a useful metric check. Missing activity is not success. These prompt preferences do not change execution, schema defaults, or final outcomes.diff --git a/frontend/snapshots.yml b/frontend/snapshots.ymlindex 7d6d5b57192c..8395a17b3301 100644--- a/frontend/snapshots.yml+++ b/frontend/snapshots.yml@@ -14862,6 +14862,10 @@ snapshots:hash: v1.k794b7964.62d15835041cea269d599d42d6f654f2e9ed4ae1e39c3f3b5eddd5833fca6394.AcjuW_1a3hPiT3q-Bn7FtXcctGRSm87IsWJU0NTgPK4scenes-app-signals-artefactloglist--linked-report-gates--light:hash: v1.k794b7964.859a100531c36c4b874e64d2bd0b2500476c8b922f8cdeee1462d1d5d6804506.R61Z2ikhPeSQuqB9dx5_VB6HF-IAjCuhVo932jK6Rt8+ scenes-app-signals-artefactloglist--monitoring-observations--dark:+ hash: v1.k794b7964.61aeca618751db24875158433b2c50b315129b5b1dc623b548126e8e6508288d.4Gs7G4G9Dpk5U7WahLriqVmRrS5788w-9EXhiXDjBaI+ scenes-app-signals-artefactloglist--monitoring-observations--light:+ hash: v1.k794b7964.084ff7ee7f202ca545d69ffbfad30dae0d190d2730a1cd5d1df0a506995cebb3.QDffrE9QrEwR-1MQQy1CHukEst_6ta5eG_CU86OB6CQscenes-app-signals-artefactloglist--narrow--dark:hash: v1.k794b7964.e37229f035e4588fc3b5357c5f2bf501f20d4f0c649463c8ec84179e98fc6f01.NrmQFdBbUe59pKfkhORUaSKwishZZjZf2KXinLIsK6Iscenes-app-signals-artefactloglist--narrow--light:diff --git a/products/product_analytics/backend/facade/queries.py b/products/product_analytics/backend/facade/queries.pyindex 81fe0ff442e8..ff5e8e0d1337 100644--- a/products/product_analytics/backend/facade/queries.py+++ b/products/product_analytics/backend/facade/queries.py@@ -16,6 +16,8 @@ from __future__ import annotationsfrom datetime import datetimefrom typing import TYPE_CHECKING, Any+from posthog.clickhouse.workload import Workload+from products.product_analytics.backend.facade.contracts import TrendsQueryRunResult_B = "products.product_analytics.backend.hogql_queries."@@ -83,6 +85,7 @@ def run_cached_trends_query(max_execution_time_seconds: int,cache_age_seconds: int,ch_user: ClickHouseUser | None = None,+ workload: Workload = Workload.DEFAULT,) -> TrendsQueryRunResult:"""Run a Trends query through the blocking recent-cache path."""@@ -98,6 +101,7 @@ def run_cached_trends_query(team=team,hogql_settings=HogQLGlobalSettings(max_execution_time=max_execution_time_seconds),ch_user=ch_user,+ workload=workload,)runner.is_query_service = is_api_key_access_method(get_query_tag_value("access_method"))response = runner.run(diff --git a/products/signals/backend/artefact_schemas.py b/products/signals/backend/artefact_schemas.pyindex 08d23e74f9c2..e977bb852920 100644--- a/products/signals/backend/artefact_schemas.py+++ b/products/signals/backend/artefact_schemas.py@@ -29,7 +29,8 @@ from uuid import UUIDfrom pydantic import BaseModel, ConfigDict, Field, RootModel, ValidationError, field_validator, model_validatorfrom products.signals.backend.enums import ReportLinkKind, ReportPriority, SuggestedSourceProduct-from products.signals.backend.report_checks import CheckInconclusiveReason, CheckOutcome+from products.signals.backend.report_checks import CheckComparison, CheckInconclusiveReason, CheckOutcome+from products.signals.backend.report_metrics import MAX_METRIC_UNIT_LENGTH, ReportMetricValueFormatfrom products.tasks.backend.facade.repo_selection_types import RepoSelectionResult# Product / type identifier parts must be routing-safe — mirrors the custom-agent identifier@@ -1034,6 +1035,34 @@ class CheckResult(BaseModel):return self+class CheckObservation(BaseModel):+ explanation: str+ check_id: str+ title: str+ monitoring_started_at: datetime+ measured_at: datetime+ monitoring_ends_at: datetime | None = None+ value_format: ReportMetricValueFormat | None = None+ unit: str | None = Field(default=None, max_length=MAX_METRIC_UNIT_LENGTH)+ comparison: CheckComparison | None = None+ observed_value: float | None = Field(default=None, allow_inf_nan=False)+ matches: bool | None = None+ error: str | None = None++ @model_validator(mode="after")+ def measurement_is_complete(self) -> CheckObservation:+ if self.measured_at <= self.monitoring_started_at:+ raise ValueError("An observation must end after Monitoring starts.")+ if self.monitoring_ends_at is not None and self.measured_at >= self.monitoring_ends_at:+ raise ValueError("An interim observation must precede the Monitoring deadline.")+ if self.error is None:+ if self.observed_value is None or self.matches is None or self.comparison is None:+ raise ValueError("A measured observation needs its value, comparison, and match result.")+ elif not self.error.strip() or self.observed_value is not None or self.matches is not None:+ raise ValueError("An errored observation needs an error and no measured result.")+ return self++class CheckLifecycleEntry(BaseModel):"""What every entry in a check's life carries, so the three cannot drift apart.@@ -1174,6 +1203,7 @@ LogArtefactContent = (| WorkRelease| PullRequestLink| CheckResult+ | CheckObservation| CheckScheduled| CheckExpired| CheckCancelled@@ -1210,6 +1240,7 @@ ARTEFACT_CONTENT_SCHEMAS: Mapping[str, type[BaseModel]] = {"work_release": WorkRelease,"pull_request": PullRequestLink,"check_result": CheckResult,+ "check_observation": CheckObservation,"check_scheduled": CheckScheduled,"check_expired": CheckExpired,"check_cancelled": CheckCancelled,@@ -1255,6 +1286,7 @@ NON_WRITABLE_ARTEFACT_TYPES: frozenset[str] = frozenset("work_release","pull_request","check_result",+ "check_observation","check_scheduled","check_expired","check_cancelled",diff --git a/products/signals/backend/models.py b/products/signals/backend/models.pyindex b803b3b598ce..4baf9fb15a8e 100644--- a/products/signals/backend/models.py+++ b/products/signals/backend/models.py@@ -1366,6 +1366,7 @@ class SignalReportArtefact(UUIDModel):WORK_RELEASE = "work_release"PULL_REQUEST = "pull_request"CHECK_RESULT = "check_result"+ CHECK_OBSERVATION = "check_observation"CHECK_SCHEDULED = "check_scheduled"CHECK_EXPIRED = "check_expired"CHECK_CANCELLED = "check_cancelled"@@ -1433,6 +1434,7 @@ class SignalReportArtefact(UUIDModel):ArtefactType.WORK_RELEASE,ArtefactType.PULL_REQUEST,ArtefactType.CHECK_RESULT,+ ArtefactType.CHECK_OBSERVATION,ArtefactType.CHECK_SCHEDULED,ArtefactType.CHECK_EXPIRED,ArtefactType.CHECK_CANCELLED,@@ -1507,7 +1509,7 @@ class SignalReportArtefact(UUIDModel):]@classmethod- def counts_by_report(cls, report_ids: list[str]) -> dict[str, int]:+ def counts_by_report(cls, report_ids: list[str], *, monitoring_enabled: bool = True) -> dict[str, int]:"""How many artefacts each report has, in one grouped query over the page.The inbox list renders this count for every row it returns. A correlated subquery makes@@ -1519,7 +1521,10 @@ class SignalReportArtefact(UUIDModel):return {}rows = (cls.objects.filter(report_id__in=report_ids)- .exclude(type__in=cls.SYSTEM_SCORING_ARTEFACT_TYPES)+ .exclude(+ type__in=cls.SYSTEM_SCORING_ARTEFACT_TYPES+ | ({cls.ArtefactType.CHECK_OBSERVATION} if not monitoring_enabled else set())+ ).values("report_id").annotate(artefact_count=models.Count("*")))diff --git a/products/signals/backend/report_check_agent.py b/products/signals/backend/report_check_agent.pyindex 0e14500239a5..a92236b3cd3f 100644--- a/products/signals/backend/report_check_agent.py+++ b/products/signals/backend/report_check_agent.py@@ -443,7 +443,8 @@ def run_agent_check(check: SignalReportCheck, *, now: datetime | None = None) -># this module reaches only when a check is actually due (see the same deferral in# `scout_harness/views.py`).from products.signals.backend.report_check_authoring import cancel_check # noqa: PLC0415- from products.signals.backend.report_check_execution import CheckVerdict, record_check_verdict # noqa: PLC0415+ from products.signals.backend.report_check_evaluation import CheckVerdict # noqa: PLC0415+ from products.signals.backend.report_check_execution import record_check_verdict # noqa: PLC0415from products.signals.backend.temporal.agentic.scout_scheduler import start_check_signals_scout_run # noqa: PLC0415now = now or timezone.now()diff --git a/products/signals/backend/report_check_authoring.py b/products/signals/backend/report_check_authoring.pyindex b3c26719d5ce..3a982601f272 100644--- a/products/signals/backend/report_check_authoring.py+++ b/products/signals/backend/report_check_authoring.py@@ -21,6 +21,7 @@ from functools import partialfrom typing import Literalfrom django.db import transaction+from django.db.models import Ffrom django.utils import timezoneimport structlog@@ -32,6 +33,7 @@ from products.signals.backend.models import SignalActorKind, SignalReport, Signafrom products.signals.backend.report_check_agent import agent_check_lane_availablefrom products.signals.backend.report_check_artefacts import write_check_cancelled, write_check_scheduledfrom products.signals.backend.report_check_execution import resolve_check_query+from products.signals.backend.report_check_monitoring import MONITORING_ERROR_RETRY_AFTERfrom products.signals.backend.report_check_research import research_can_reconcile_checksfrom products.signals.backend.report_check_telemetry import capture_report_check_created, capture_report_check_skippedfrom products.signals.backend.report_check_timing import metric_check_ready_at@@ -45,6 +47,8 @@ from products.signals.backend.report_checks import (MetricThresholdConfig,authored_monitoring_interval_minutes,check_schedule_expires_at,+ monitoring_interval_minutes,+ monitoring_window_end,parse_check_config,soak_minutes_from_gap,validate_metric_check_for_write,@@ -119,7 +123,9 @@ def create_check(else now + timedelta(minutes=soak_minutes))metric_ready_at = None- if kind == SignalReportCheck.Kind.METRIC_THRESHOLD:+ if kind == SignalReportCheck.Kind.METRIC_THRESHOLD and not (+ stored_config.get("monitoring") and locked_report.effective_status != SignalReport.Status.RESOLVED+ ):metric_ready_at = metric_check_ready_at(stored_config["query"], locked_report.team, now)first_run_at = max(first_run_at, metric_ready_at)last_run_at = first_run_at + timedelta(minutes=(run_interval_minutes or 0) * max(0, runs_remaining - 1))@@ -508,7 +514,9 @@ def cancel_check(return bool(cancelled)-def _arm_pending_checks(*, team_id: int, report_id: str | uuid.UUID, resolved_at: datetime, armed_at: datetime) -> int:+def _arm_pending_checks(+ *, team_id: int, report_id: str | uuid.UUID, resolved_at: datetime, armed_at: datetime, monitoring: bool = False+) -> int:"""Arm rows individually because each check carries its own soak and schedule.The measurement origin can precede this write when implementation completes before the status update.@@ -522,7 +530,24 @@ def _arm_pending_checks(*, team_id: int, report_id: str | uuid.UUID, resolved_atarmed = 0for check in pending:next_run_at = resolved_at + timedelta(minutes=check.soak_minutes or 0)- if check.kind == SignalReportCheck.Kind.METRIC_THRESHOLD:+ interim = (+ monitoring+ and check.kind == SignalReportCheck.Kind.METRIC_THRESHOLD+ and bool(check.config.get("monitoring"))+ )+ resuming_monitoring = interim and check.last_run_at is not None and check.last_run_at >= resolved_at+ if interim:+ last_sample_at = max(resolved_at, check.last_run_at or resolved_at)+ delay = (+ MONITORING_ERROR_RETRY_AFTER+ if resuming_monitoring and check.consecutive_errors > 0+ else timedelta(minutes=monitoring_interval_minutes(check.run_interval_minutes))+ )+ next_run_at = min(+ last_sample_at + delay,+ monitoring_window_end(resolved_at, check.soak_minutes),+ )+ elif check.kind == SignalReportCheck.Kind.METRIC_THRESHOLD:try:config = parse_check_config(check.kind, check.config)assert isinstance(config, MetricThresholdConfig)@@ -531,11 +556,15 @@ def _arm_pending_checks(*, team_id: int, report_id: str | uuid.UUID, resolved_atexcept (CheckConfigValidationError, ValueError):# An invalid legacy config must not prevent the report's other checks from arming.pass- expires_at = check_schedule_expires_at(- next_run_at=next_run_at,- run_interval_minutes=check.run_interval_minutes,- runs_remaining=check.runs_remaining,- start_at=resolved_at,+ expires_at = (+ resolved_at + MAX_CHECK_HORIZON+ if interim+ else check_schedule_expires_at(+ next_run_at=next_run_at,+ run_interval_minutes=check.run_interval_minutes,+ runs_remaining=check.runs_remaining,+ start_at=resolved_at,+ ))armed += (SignalReportCheck.objects.for_team(team_id)@@ -545,6 +574,7 @@ def _arm_pending_checks(*, team_id: int, report_id: str | uuid.UUID, resolved_atnext_run_at=next_run_at,measurement_start_at=resolved_at if check.kind == SignalReportCheck.Kind.METRIC_THRESHOLD else None,expires_at=expires_at,+ consecutive_errors=0 if interim and not resuming_monitoring else F("consecutive_errors"),updated_at=armed_at,))@@ -576,4 +606,10 @@ def arm_pending_checks(armed_at = resolved_atelse:return 0- return _arm_pending_checks(team_id=team_id, report_id=report_id, resolved_at=resolved_at, armed_at=armed_at)+ return _arm_pending_checks(+ team_id=team_id,+ report_id=report_id,+ resolved_at=resolved_at,+ armed_at=armed_at,+ monitoring=status == SignalReport.Status.MONITORING,+ )diff --git a/products/signals/backend/report_check_evaluation.py b/products/signals/backend/report_check_evaluation.pynew file mode 100644index 000000000000..8e9e492cdc92--- /dev/null+++ b/products/signals/backend/report_check_evaluation.py@@ -0,0 +1,99 @@+from __future__ import annotations++from posthog.schema import (+ AlertCondition,+ AlertConditionType,+ InsightsThresholdBounds,+ InsightThreshold,+ InsightThresholdType,+)++from posthog.dataclasses import frozen++from products.alerts.backend.facade.evaluation import (+ ComparableSeries,+ ExtractionResult,+ SeriesPoint,+ evaluate_threshold,+)+from products.signals.backend.report_checks import CheckComparison, CheckInconclusiveReason, CheckOutcome++MAX_CHECK_ERROR_REASON_LENGTH = 300+++@frozen+class CheckVerdict:+ outcome: CheckOutcome+ explanation: str+ observed_value: float | None = None+ # Required on an `inconclusive` verdict and refused on any other, so a stored reason always+ # explains the outcome next to it.+ reason: CheckInconclusiveReason | None = None++ def __post_init__(self) -> None:+ if (self.outcome == "inconclusive") != (self.reason is not None):+ raise ValueError("an `inconclusive` verdict needs a reason, and no other verdict takes one")+++def _threshold_for(comparison: CheckComparison) -> InsightThreshold:+ """The alerts threshold whose breach is exactly this comparison's failure.++ The alerts comparator breaches on `value < lower` or `value > upper`, so `gte` is a lower bound,+ `lte` an upper one, and `between` both.+ """+ if comparison.operator == "between":+ assert comparison.bounds is not None+ bounds = InsightsThresholdBounds(lower=comparison.bounds.lower, upper=comparison.bounds.upper)+ elif comparison.operator == "gte":+ bounds = InsightsThresholdBounds(lower=comparison.value)+ else:+ bounds = InsightsThresholdBounds(upper=comparison.value)+ return InsightThreshold(type=InsightThresholdType.ABSOLUTE, bounds=bounds)+++def describe_comparison(comparison: CheckComparison) -> str:+ if comparison.operator == "between":+ assert comparison.bounds is not None+ return f"between {comparison.bounds.lower} and {comparison.bounds.upper}"+ return f"at most {comparison.value}" if comparison.operator == "lte" else f"at least {comparison.value}"+++def evaluate_check_value(*, comparison: CheckComparison, observed_value: float, subject: str) -> CheckVerdict:+ """Compare one measured value against the check's expectation."""++ result = ExtractionResult(+ series=[+ ComparableSeries(label=subject, points=[SeriesPoint(date=None, value=observed_value)], current_index=0)+ ],+ subject=subject,+ framed=False,+ )+ evaluation = evaluate_threshold(+ result,+ AlertCondition(type=AlertConditionType.ABSOLUTE_VALUE),+ _threshold_for(comparison),+ )+ if evaluation.breaches:+ return CheckVerdict(outcome="failed", explanation=evaluation.breaches[0], observed_value=observed_value)+ return CheckVerdict(+ outcome="passed",+ explanation=f"{subject} ({observed_value}) is {describe_comparison(comparison)}, as expected",+ observed_value=observed_value,+ )+++def errored_check_explanation(error: Exception, *, subject: str) -> str:+ """The line a report reader sees when a run could not be measured.++ It opens with the check's title, as the passed and failed lines do, so a report carrying several+ checks does not log entries a reader cannot tell apart.++ Our own validation and resolution failures name something the reader can act on, so they are+ kept. Anything else is reduced to the fixed line: a query error can carry generated SQL and a+ server stack trace, which `ExposedCHQueryError` exists to keep out of user-facing text, and the+ artefact log is permanent and rendered as written. Measurement callers log the whole error.+ """+ reason = str(error).strip() if isinstance(error, ValueError | TimeoutError) else ""+ if not reason:+ return f"{subject} could not be measured."+ return f"{subject} could not be measured: {reason[:MAX_CHECK_ERROR_REASON_LENGTH]}"diff --git a/products/signals/backend/report_check_execution.py b/products/signals/backend/report_check_execution.pyindex ebd5208c495a..905f1edbd732 100644--- a/products/signals/backend/report_check_execution.py+++ b/products/signals/backend/report_check_execution.py@@ -6,8 +6,8 @@ comparison, one artefact, with no sandbox, no scout enrolment, and no LLM. An ``a run to decide it, so it is handed to ``report_check_agent`` and closed later by the tool that runcalls.-Both lanes end in ``record_check_verdict``, which is the only writer of a check's outcome: one-artefact appended and one row advanced or retired, whatever decided the verdict.+Terminal results go through ``record_check_verdict``. Monitoring metric observations use+``report_check_monitoring`` to save measurements without concluding the check or report.The comparison is the alerts product's, not a second one written here, so a check and an alert worda breach the same way and there is one place where "is this value out of bounds?" is decided.@@ -23,36 +23,45 @@ from typing import castfrom uuid import UUIDfrom django.db import transaction-from django.db.models import Exists, F, OuterRef, Q, Window-from django.db.models.functions import RowNumber+from django.db.models import (+ Case,+ DateTimeField,+ DurationField,+ Exists,+ ExpressionWrapper,+ F,+ OuterRef,+ Q,+ Value,+ When,+ Window,+)+from django.db.models.functions import Coalesce, Greatest, Least, RowNumberfrom django.utils import timezoneimport structlog-from posthog.schema import (- AlertCondition,- AlertConditionType,- InsightsThresholdBounds,- InsightThreshold,- InsightThresholdType,-)-from posthog.clickhouse.client.connection import ClickHouseUserfrom posthog.clickhouse.query_tagging import tag_queriesfrom posthog.dataclasses import frozenfrom posthog.models import Team-from products.alerts.backend.facade.evaluation import (- ComparableSeries,- ExtractionResult,- SeriesPoint,- evaluate_threshold,-)from products.signals.backend.artefact_attribution import ArtefactAttributionfrom products.signals.backend.artefact_schemas import CheckResultfrom products.signals.backend.enums import SignalSourceProduct, SignalSourceTypefrom products.signals.backend.models import SignalReport, SignalReportArtefact, SignalReportCheckfrom products.signals.backend.report_check_artefacts import write_check_expired+from products.signals.backend.report_check_evaluation import (+ CheckVerdict,+ describe_comparison,+ errored_check_explanation,+ evaluate_check_value,+)+from products.signals.backend.report_check_monitoring import (+ MonitoringWindowClosed,+ measure_monitoring_check,+ record_monitoring_observation,+)from products.signals.backend.report_check_telemetry import (capture_report_check_evaluated,capture_report_checks_expired,@@ -61,19 +70,19 @@ from products.signals.backend.report_check_timing import metric_check_ready_at,from products.signals.backend.report_checks import (AWAITING_DATA_RETRY_WAITS,DEFAULT_CHECK_SOAK_HOURS,+ DEFAULT_MONITORING_INTERVAL_MINUTES,MAX_CHECK_HORIZON,MAX_CONSECUTIVE_CHECK_ERRORS,- CheckComparison,+ MIN_CHECK_INTERVAL_MINUTES,CheckConfigValidationError,- CheckInconclusiveReason,- CheckOutcome,MetricThresholdConfig,check_schedule_expires_at,parse_check_config,soak_minutes_from_gap,)+from products.signals.backend.report_content_gates import team_report_monitoring_enabledfrom products.signals.backend.report_lifecycle import check_monitoring_disabled_organizations, report_status_filter-from products.signals.backend.report_metric_refresh import measure_metric+from products.signals.backend.report_metric_refresh import MetricQueryBudgetExhausted, measure_metricfrom products.signals.backend.report_metrics import validate_live_metric_querylogger = structlog.get_logger(__name__)@@ -89,8 +98,6 @@ MAX_CHECK_EXPIRIES_PER_TICK = 500# An errored run retries on the next window instead of retiring the check, so a transient query# failure does not end a soak. It does not consume `runs_remaining`.CHECK_ERROR_RETRY_AFTER = timedelta(hours=6)-# Same bound the scout runner puts on a stored failure reason.-MAX_CHECK_ERROR_REASON_LENGTH = 300# Weight of the signal a failed check on a resolved report emits. High enough to promote on its own:# a fix that stopped holding is a finding somebody already decided was worth fixing once.CHECK_FAILURE_SIGNAL_WEIGHT = 1.0@@ -106,6 +113,7 @@ class CheckRunSummary:failed: interrored: intinconclusive: int = 0+ observed: int = 0# An `agent` check the tick started a scout run for, and one the fleet could not take yet.# Neither is an outcome: both rows are still active and still owe a verdict.dispatched: int = 0@@ -114,20 +122,6 @@ class CheckRunSummary:cancelled: int = 0-@frozen-class CheckVerdict:- outcome: CheckOutcome- explanation: str- observed_value: float | None = None- # Required on an `inconclusive` verdict and refused on any other, so a stored reason always- # explains the outcome next to it.- reason: CheckInconclusiveReason | None = None-- def __post_init__(self) -> None:- if (self.outcome == "inconclusive") != (self.reason is not None):- raise ValueError("an `inconclusive` verdict needs a reason, and no other verdict takes one")--@frozenclass _CheckTransition:"""Where a check goes after one verdict. `next_run_at` is None when the check retires."""@@ -160,71 +154,6 @@ def resolve_check_query(config: MetricThresholdConfig, report: SignalReport) ->raise ValueError(f"the report has no metric `{config.metric_id}` to measure")-def _threshold_for(comparison: CheckComparison) -> InsightThreshold:- """The alerts threshold whose breach is exactly this comparison's failure.-- The alerts comparator breaches on `value < lower` or `value > upper`, so `gte` is a lower bound,- `lte` an upper one, and `between` both.- """- if comparison.operator == "between":- assert comparison.bounds is not None- bounds = InsightsThresholdBounds(lower=comparison.bounds.lower, upper=comparison.bounds.upper)- elif comparison.operator == "gte":- bounds = InsightsThresholdBounds(lower=comparison.value)- else:- bounds = InsightsThresholdBounds(upper=comparison.value)- return InsightThreshold(type=InsightThresholdType.ABSOLUTE, bounds=bounds)---def _describe_comparison(comparison: CheckComparison) -> str:- if comparison.operator == "between":- assert comparison.bounds is not None- return f"between {comparison.bounds.lower} and {comparison.bounds.upper}"- return f"at most {comparison.value}" if comparison.operator == "lte" else f"at least {comparison.value}"---def evaluate_check_value(*, comparison: CheckComparison, observed_value: float, subject: str) -> CheckVerdict:- """Compare one measured value against the check's expectation."""-- result = ExtractionResult(- series=[- ComparableSeries(label=subject, points=[SeriesPoint(date=None, value=observed_value)], current_index=0)- ],- subject=subject,- framed=False,- )- evaluation = evaluate_threshold(- result,- AlertCondition(type=AlertConditionType.ABSOLUTE_VALUE),- _threshold_for(comparison),- )- if evaluation.breaches:- return CheckVerdict(outcome="failed", explanation=evaluation.breaches[0], observed_value=observed_value)- return CheckVerdict(- outcome="passed",- explanation=f"{subject} ({observed_value}) is {_describe_comparison(comparison)}, as expected",- observed_value=observed_value,- )---def _errored_explanation(error: Exception, *, subject: str) -> str:- """The line a report reader sees when a run could not be measured.-- It opens with the check's title, as the passed and failed lines do, so a report carrying several- checks does not log entries a reader cannot tell apart.-- Our own validation and resolution failures name something the reader can act on, so they are- kept. Anything else is reduced to the fixed line: a query error can carry generated SQL and a- server stack trace, which `ExposedCHQueryError` exists to keep out of user-facing text, and the- artefact log is permanent and rendered as written. `logger.exception` above holds the whole- error either way.- """- reason = str(error).strip() if isinstance(error, ValueError | TimeoutError) else ""- if not reason:- return f"{subject} could not be measured."- return f"{subject} could not be measured: {reason[:MAX_CHECK_ERROR_REASON_LENGTH]}"--def measure_check(check: SignalReportCheck, *, deadline: float) -> CheckVerdict:"""Run one check and return its verdict. Never raises: a failure to measure is an `errored` verdict."""@@ -258,7 +187,7 @@ def measure_check(check: SignalReportCheck, *, deadline: float) -> CheckVerdict:)except Exception as error:logger.exception("signals.report_check.measurement_failed", check_id=str(check.id), team_id=check.team_id)- return CheckVerdict(outcome="errored", explanation=_errored_explanation(error, subject=check.title))+ return CheckVerdict(outcome="errored", explanation=errored_check_explanation(error, subject=check.title))return evaluate_check_value(comparison=config.comparison, observed_value=measurement.value, subject=check.title)@@ -409,14 +338,14 @@ def record_check_verdict(reason=verdict.reason,observed_value=verdict.observed_value,baseline_value=config.baseline_value if config is not None else None,- threshold=_describe_comparison(config.comparison) if config is not None else None,+ threshold=describe_comparison(config.comparison) if config is not None else None,run_id=run_id,),attribution=attribution or ArtefactAttribution.system(),)if _should_resurface(current, verdict, lane_can_emit_report=lane_can_emit_report):baseline = config.baseline_value if config is not None else None- threshold = _describe_comparison(config.comparison) if config is not None else None+ threshold = describe_comparison(config.comparison) if config is not None else Nonetransaction.on_commit(lambda: resurface_failed_check(current, verdict, baseline, threshold))# Post-commit, so a rolled-back verdict is never counted as one, and after the artefact so# the event describes a result a reader can already see on the report. The team comes off@@ -508,17 +437,41 @@ def resurface_failed_check(logger.exception("signals.report_check.resurface_failed", check_id=str(check.id), team_id=check.team_id)-def expire_overdue_checks(now: datetime) -> int:+def _monitoring_metric_filter(disabled_organizations: frozenset[UUID], *, started_before: datetime | None = None) -> Q:+ reports = SignalReport.objects.filter(+ id=OuterRef("report_id"),+ team_id=OuterRef("team_id"),+ status=SignalReport.Status.MONITORING,+ monitoring_started_at__isnull=False,+ ).exclude(team__organization_id__in=disabled_organizations)+ if started_before is not None:+ reports = reports.filter(monitoring_started_at__lte=started_before)+ # Joined UPDATE filters become an ID subquery, so PostgreSQL cannot recheck stale check guards after a lock wait.+ return Q(Exists(reports), kind=SignalReportCheck.Kind.METRIC_THRESHOLD, config__monitoring__isnull=False) & ~Q(+ config__monitoring=None+ )+++def expire_overdue_checks(now: datetime, *, disabled_monitoring_organization_ids: frozenset[UUID] | None = None) -> int:"""Retire open checks whose horizon passed without a run. Returns how many were retired.Pending rows are swept too, so a check waiting on a report that never resolves retires at thehorizon instead of waiting forever for a clock that will not start."""+ expiry_filter = Q(expires_at__lte=now)+ if disabled_monitoring_organization_ids is not None:+ monitoring = _monitoring_metric_filter(disabled_monitoring_organization_ids) & Q(+ status=SignalReportCheck.Status.ACTIVE+ )+ expiry_filter = (expiry_filter & ~monitoring) | (+ _monitoring_metric_filter(disabled_monitoring_organization_ids, started_before=now - MAX_CHECK_HORIZON)+ & Q(status=SignalReportCheck.Status.ACTIVE)+ )overdue = list(# Only the columns the log entry and the telemetry read. A `metric_threshold` config carries# a copied-in query, and a full tick hydrates five hundred rows.- SignalReportCheck.all_teams.filter(status__in=SignalReportCheck.OPEN_STATUSES, expires_at__lte=now).only(+ SignalReportCheck.all_teams.filter(expiry_filter, status__in=SignalReportCheck.OPEN_STATUSES).only("id", "team_id", "report_id", "kind", "title", "last_run_at")[:MAX_CHECK_EXPIRIES_PER_TICK])@@ -528,9 +481,9 @@ def expire_overdue_checks(now: datetime) -> int:# which arms its pending checks and moves `expires_at` forward, and an update filtered on id and# status alone would retire a check that has just been given a clock.expired = SignalReportCheck.all_teams.filter(+ expiry_filter,id__in=[check.id for check in overdue],status__in=SignalReportCheck.OPEN_STATUSES,- expires_at__lte=now,).update(status=SignalReportCheck.Status.EXPIRED, updated_at=now)if expired:_log_expired_checks(overdue, now, expired)@@ -587,6 +540,7 @@ def _report_expired_checks(overdue: list[SignalReportCheck]) -> None:def _park_checks(active: Iterable[SignalReportCheck], now: datetime, *, resolved_status_filter: Q | None = None) -> int:parked = 0team_filters: dict[int, Q] = {}+ monitoring_enabled: dict[int, bool] = {}for check in active:status_filter = resolved_status_filterif status_filter is None:@@ -605,6 +559,16 @@ def _park_checks(active: Iterable[SignalReportCheck], now: datetime, *, resolvedif check.last_run_at is not None or check.dispatched_at is not Noneelse soak_minutes_from_gap(check.next_run_at, check.created_at))+ preserve_monitoring_errors = False+ if (+ check.kind == SignalReportCheck.Kind.METRIC_THRESHOLD+ and check.config.get("monitoring")+ and check.measurement_start_at is not None+ and check.measurement_start_at == check.report.monitoring_started_at+ ):+ if check.team_id not in monitoring_enabled:+ monitoring_enabled[check.team_id] = team_report_monitoring_enabled(check.team_id)+ preserve_monitoring_errors = monitoring_enabled[check.team_id]parked += (SignalReportCheck.objects.for_team(check.team_id).filter(id=check.id, status=SignalReportCheck.Status.ACTIVE)@@ -615,7 +579,7 @@ def _park_checks(active: Iterable[SignalReportCheck], now: datetime, *, resolvedmeasurement_start_at=None,next_run_at=now + timedelta(minutes=soak_minutes),expires_at=now + MAX_CHECK_HORIZON,- consecutive_errors=0,+ consecutive_errors=F("consecutive_errors") if preserve_monitoring_errors else 0,consecutive_inconclusive=0,dispatched_at=None,updated_at=now,@@ -626,7 +590,9 @@ def _park_checks(active: Iterable[SignalReportCheck], now: datetime, *, resolveddef park_report_checks_on_reopen(*, team_id: int, report_id: str, now: datetime) -> int:return _park_checks(- SignalReportCheck.objects.for_team(team_id).filter(report_id=report_id, status=SignalReportCheck.Status.ACTIVE),+ SignalReportCheck.objects.for_team(team_id)+ .filter(report_id=report_id, status=SignalReportCheck.Status.ACTIVE)+ .select_related("report"),now,)@@ -643,8 +609,8 @@ def park_checks_on_unresolved_reports(through `arm_pending_checks` with a fresh soak, and the expiry sweep retires it if the reportnever resolves.- A dated row keeps the gap its author left as its soak. The error streak and any open dispatch- are cleared, so retry accounting starts again after the next resolve.+ A dated row keeps the gap its author left as its soak. An interrupted Monitoring period keeps+ its error streak so restoring it cannot postpone a retry or reset the error budget."""candidates = SignalReportCheck.all_teams.filter(status=SignalReportCheck.Status.ACTIVE)if disabled_monitoring_organization_ids is None:@@ -655,8 +621,20 @@ def park_checks_on_unresolved_reports(disabled_monitoring_organization_ids=disabled_monitoring_organization_ids,)active = list(- candidates.exclude(status_filter).only(- "id", "team_id", "created_at", "next_run_at", "soak_minutes", "last_run_at", "dispatched_at"+ candidates.exclude(status_filter)+ .select_related("report")+ .only(+ "id",+ "team_id",+ "created_at",+ "next_run_at",+ "soak_minutes",+ "last_run_at",+ "dispatched_at",+ "kind",+ "config",+ "measurement_start_at",+ "report__monitoring_started_at",)[:MAX_CHECK_PARKS_PER_TICK])parked = _park_checks(active, now, resolved_status_filter=status_filter)@@ -665,6 +643,81 @@ def park_checks_on_unresolved_reports(return parked+def _monitoring_window_end_expression() -> ExpressionWrapper:+ return ExpressionWrapper(+ F("report__monitoring_started_at") + Coalesce("soak_minutes", Value(0)) * Value(timedelta(minutes=1)),+ output_field=DateTimeField(),+ )+++def _prepare_monitoring_metrics(now: datetime, disabled_organizations: frozenset[UUID]) -> None:+ cadence = ExpressionWrapper(+ Greatest(+ Coalesce("run_interval_minutes", Value(DEFAULT_MONITORING_INTERVAL_MINUTES)),+ Value(MIN_CHECK_INTERVAL_MINUTES),+ )+ * Value(timedelta(minutes=1)),+ output_field=DurationField(),+ )+ candidates = (+ SignalReportCheck.all_teams.filter(+ _monitoring_metric_filter(disabled_organizations),+ status=SignalReportCheck.Status.ACTIVE,+ )+ .annotate(_monitoring_ends_at=_monitoring_window_end_expression())+ .annotate(+ _monitoring_next_run_at=Least(+ F("_monitoring_ends_at"),+ ExpressionWrapper(+ Greatest(+ Coalesce("last_run_at", F("report__monitoring_started_at")), F("report__monitoring_started_at")+ )+ + Case(+ When(+ last_run_at__gte=F("report__monitoring_started_at"),+ consecutive_errors__gt=0,+ then=Value(CHECK_ERROR_RETRY_AFTER),+ ),+ default=cadence,+ output_field=DurationField(),+ ),+ output_field=DateTimeField(),+ ),+ ),+ _monitoring_expires_at=ExpressionWrapper(+ F("report__monitoring_started_at") + Value(MAX_CHECK_HORIZON), output_field=DateTimeField()+ ),+ )+ .exclude(+ measurement_start_at=F("report__monitoring_started_at"),+ next_run_at=F("_monitoring_next_run_at"),+ expires_at=F("_monitoring_expires_at"),+ )+ .select_related("report")+ .order_by("report__monitoring_started_at", "id")[:MAX_CHECK_PARKS_PER_TICK]+ )+ for check in candidates:+ start = check.report.monitoring_started_at+ assert start is not None+ period = SignalReport.objects.filter(+ id=OuterRef("report_id"),+ team_id=OuterRef("team_id"),+ status=SignalReport.Status.MONITORING,+ monitoring_started_at=start,+ )+ SignalReportCheck.objects.for_team(check.team_id).filter(+ Exists(period),+ id=check.id,+ status=SignalReportCheck.Status.ACTIVE,+ updated_at=check.updated_at,+ ).update(+ next_run_at=check._monitoring_next_run_at,+ measurement_start_at=start,+ expires_at=check._monitoring_expires_at,+ updated_at=now,+ )++def collect_due_checks(now: datetime,*,@@ -684,15 +737,18 @@ def collect_due_checks(if disabled_monitoring_organization_ids is None:disabled_monitoring_organization_ids = check_monitoring_disabled_organizations(eligible_checks)candidates = (- eligible_checks.filter(+ eligible_checks.annotate(_monitoring_ends_at=_monitoring_window_end_expression())+ .filter(report_status_filter([SignalReport.Status.RESOLVED],field="report__status",disabled_monitoring_organization_ids=disabled_monitoring_organization_ids,+ )+ | (+ _monitoring_metric_filter(disabled_monitoring_organization_ids)+ & Q(report__monitoring_started_at__gt=now - MAX_CHECK_HORIZON, _monitoring_ends_at__gt=now)),- # A check re-measures a fix, so it runs only while its report is resolved. The park step- # moves any other active row back to `pending`; this filter keeps a row it has not reached- # yet out of the tick.+ # Agent and legacy metric checks must stay dormant until the report resolves.).select_related("report", "report__team", "team__organization")# Rank each team's rows against its own, then read those ranks in order, so every team's@@ -759,15 +815,17 @@ def run_due_report_checks(*, now: datetime | None = None, limit: int = MAX_CHECK"""now = now or timezone.now()- expired = expire_overdue_checks(now)candidate_checks = SignalReportCheck.all_teams.filter(Q(status=SignalReportCheck.Status.ACTIVE) | Q(status=SignalReportCheck.Status.PENDING, expires_at__gt=now))disabled_organizations = check_monitoring_disabled_organizations(candidate_checks)+ _prepare_monitoring_metrics(now, disabled_organizations)+ expired = expire_overdue_checks(now, disabled_monitoring_organization_ids=disabled_organizations)_arm_monitoring_checks(now, disabled_monitoring_organization_ids=disabled_organizations)park_checks_on_unresolved_reports(now, disabled_monitoring_organization_ids=disabled_organizations)deadline = time.monotonic() + CHECK_RUN_TIME_BUDGET_SECONDScounts: dict[str, int] = {+ "observed": 0,"passed": 0,"failed": 0,"errored": 0,@@ -779,6 +837,23 @@ def run_due_report_checks(*, now: datetime | None = None, limit: int = MAX_CHECKfor check in collect_due_checks(now, limit=limit, disabled_monitoring_organization_ids=disabled_organizations):if time.monotonic() >= deadline:break+ if (+ check.report.status == SignalReport.Status.MONITORING+ and check.team.organization_id not in disabled_organizations+ and check.kind == SignalReportCheck.Kind.METRIC_THRESHOLD+ and check.config.get("monitoring")+ and check.report.monitoring_started_at is not None+ ):+ try:+ observation = measure_monitoring_check(check, now=now, deadline=deadline)+ counts["observed"] += int(record_monitoring_observation(check, observation))+ except (MetricQueryBudgetExhausted, MonitoringWindowClosed):+ counts["deferred"] += 1+ except Exception:+ logger.exception(+ "signals.report_check.monitoring_step_failed", check_id=str(check.id), team_id=check.team_id+ )+ continueif check.kind == SignalReportCheck.Kind.AGENT:# Deferred: the agent lane reaches the scout harness and the Temporal client, and this# module is imported by the REST route load, which never dispatches anything.@@ -795,10 +870,11 @@ def run_due_report_checks(*, now: datetime | None = None, limit: int = MAX_CHECKif check.kind == SignalReportCheck.Kind.METRIC_THRESHOLD:start_at = check.measurement_start_at or nowtry:- query = resolve_check_query(- cast(MetricThresholdConfig, parse_check_config(check.kind, check.config)), check.report- )+ config = cast(MetricThresholdConfig, parse_check_config(check.kind, check.config))+ query = resolve_check_query(config, check.report)ready_at = metric_check_ready_at(query, check.report.team, start_at)+ if config.monitoring is not None:+ ready_at = max(ready_at, start_at + timedelta(minutes=check.soak_minutes or 0))except (CheckConfigValidationError, ValueError):ready_at = nowif check.measurement_start_at is None or now < ready_at:diff --git a/products/signals/backend/report_check_monitoring.py b/products/signals/backend/report_check_monitoring.pynew file mode 100644index 000000000000..6fdb6bc0228a--- /dev/null+++ b/products/signals/backend/report_check_monitoring.py@@ -0,0 +1,151 @@+from __future__ import annotations++from datetime import datetime, timedelta++from django.db import transaction++import structlog++from posthog.clickhouse.client.connection import ClickHouseUser+from posthog.clickhouse.workload import Workload++from products.signals.backend.artefact_attribution import ArtefactAttribution+from products.signals.backend.artefact_schemas import CheckObservation+from products.signals.backend.models import SignalReport, SignalReportArtefact, SignalReportCheck+from products.signals.backend.report_check_evaluation import errored_check_explanation, evaluate_check_value+from products.signals.backend.report_check_timing import monitoring_metric_query+from products.signals.backend.report_checks import (+ MAX_CONSECUTIVE_CHECK_ERRORS,+ MetricThresholdConfig,+ monitoring_comparison,+ monitoring_interval_minutes,+ monitoring_window_end,+ parse_check_config,+ validate_metric_check_for_write,+)+from products.signals.backend.report_content_gates import team_report_monitoring_enabled+from products.signals.backend.report_metric_refresh import MetricQueryBudgetExhausted, measure_metric++logger = structlog.get_logger(__name__)+MONITORING_ERROR_RETRY_AFTER = timedelta(hours=6)+++class MonitoringWindowClosed(Exception):+ pass+++def measure_monitoring_check(check: SignalReportCheck, *, now: datetime, deadline: float) -> CheckObservation:+ start = check.report.monitoring_started_at+ assert start is not None+ end = monitoring_window_end(start, check.soak_minutes)+ if now >= end:+ raise MonitoringWindowClosed+ comparison = None+ config = None+ try:+ config = parse_check_config(check.kind, check.config)+ assert isinstance(config, MetricThresholdConfig)+ validate_metric_check_for_write(config)+ assert config.query is not None+ comparison = monitoring_comparison(config, now - start)+ query = monitoring_metric_query(config.query, start=start, cutoff=now)+ measurement = measure_metric(+ query,+ check.report.team,+ deadline=deadline,+ include_series=False,+ ch_user=ClickHouseUser.SIGNALS,+ workload=Workload.OFFLINE,+ )+ verdict = evaluate_check_value(comparison=comparison, observed_value=measurement.value, subject=check.title)+ return CheckObservation(+ check_id=str(check.id),+ title=check.title,+ monitoring_started_at=start,+ measured_at=now,+ monitoring_ends_at=end,+ value_format=config.value_format if isinstance(config, MetricThresholdConfig) else None,+ unit=config.unit if isinstance(config, MetricThresholdConfig) else None,+ comparison=comparison,+ observed_value=measurement.value,+ matches=verdict.outcome == "passed",+ explanation=verdict.explanation,+ )+ except MetricQueryBudgetExhausted:+ raise+ except Exception as error:+ logger.exception(+ "signals.report_check.monitoring_measurement_failed", check_id=str(check.id), team_id=check.team_id+ )+ return CheckObservation(+ check_id=str(check.id),+ title=check.title,+ monitoring_started_at=start,+ measured_at=now,+ monitoring_ends_at=end,+ value_format=config.value_format if isinstance(config, MetricThresholdConfig) else None,+ unit=config.unit if isinstance(config, MetricThresholdConfig) else None,+ comparison=comparison,+ error=errored_check_explanation(error, subject=check.title),+ explanation=errored_check_explanation(error, subject=check.title),+ )+++def record_monitoring_observation(check: SignalReportCheck, observation: CheckObservation) -> bool:+ with transaction.atomic():+ report = SignalReport.objects.select_for_update().filter(id=check.report_id, team_id=check.team_id).first()+ if (+ report is None+ or report.status != SignalReport.Status.MONITORING+ or report.monitoring_started_at != observation.monitoring_started_at+ or not team_report_monitoring_enabled(check.team_id)+ ):+ return False+ current = (+ SignalReportCheck.objects.for_team(check.team_id)+ .select_for_update()+ .filter(+ id=check.id,+ report_id=report.id,+ status=SignalReportCheck.Status.ACTIVE,+ )+ .first()+ )+ if (+ current is None+ or current.updated_at != check.updated_at+ or current.config != check.config+ or current.next_run_at != check.next_run_at+ or current.last_run_at != check.last_run_at+ or current.measurement_start_at != observation.monitoring_started_at+ or current.expires_at <= observation.measured_at+ or observation.measured_at >= monitoring_window_end(observation.monitoring_started_at, current.soak_minutes)+ or (+ observation.monitoring_ends_at is not None+ and observation.monitoring_ends_at+ != monitoring_window_end(observation.monitoring_started_at, current.soak_minutes)+ )+ ):+ return False+ current.last_run_at = observation.measured_at+ if observation.error is not None:+ current.consecutive_errors += 1+ if current.consecutive_errors >= MAX_CONSECUTIVE_CHECK_ERRORS:+ current.status = SignalReportCheck.Status.ERRORED+ current.next_run_at = observation.measured_at + MONITORING_ERROR_RETRY_AFTER+ else:+ current.consecutive_errors = 0+ current.next_run_at = observation.measured_at + timedelta(+ minutes=monitoring_interval_minutes(current.run_interval_minutes)+ )+ current.next_run_at = min(+ current.next_run_at, monitoring_window_end(observation.monitoring_started_at, current.soak_minutes)+ )+ current.save(update_fields=["last_run_at", "consecutive_errors", "next_run_at", "status", "updated_at"])+ SignalReportArtefact.add_log(+ team_id=current.team_id,+ report_id=str(current.report_id),+ content=observation,+ attribution=ArtefactAttribution.system(),+ )+ return Truediff --git a/products/signals/backend/report_check_timing.py b/products/signals/backend/report_check_timing.pyindex 6d286b8caa8a..1c6a136c24af 100644--- a/products/signals/backend/report_check_timing.py+++ b/products/signals/backend/report_check_timing.py@@ -43,3 +43,21 @@ def metric_check_ready_at(query: dict[str, Any], team: Team, start_at: datetime)while metric_check_window_start(query, team, candidate) < start_at.astimezone(UTC):candidate += stepreturn candidate.astimezone(UTC)+++def monitoring_metric_query(query: dict[str, Any], *, start: datetime, cutoff: datetime) -> dict[str, Any]:+ if cutoff <= start:+ raise ValueError("A Monitoring measurement must end after Monitoring starts.")+ return {+ **query,+ "source": {+ **query["source"],+ "dateRange": {+ **query["source"]["dateRange"],+ "date_from": start.isoformat(),+ "date_to": cutoff.isoformat(),+ "explicitDate": True,+ "excludeIncompletePeriods": False,+ },+ },+ }diff --git a/products/signals/backend/report_checks.py b/products/signals/backend/report_checks.pyindex 5f3eba0a9da6..e2867afc5ad2 100644--- a/products/signals/backend/report_checks.py+++ b/products/signals/backend/report_checks.py@@ -264,6 +264,35 @@ class MetricThresholdConfig(BaseModel):return self+DEFAULT_MONITORING_INTERVAL_MINUTES = 24 * 60+++def monitoring_window_end(start: datetime, soak_minutes: int | None) -> datetime:+ return start + timedelta(minutes=soak_minutes or 0)+++def monitoring_interval_minutes(run_interval_minutes: int | None) -> int:+ return max(MIN_CHECK_INTERVAL_MINUTES, run_interval_minutes or DEFAULT_MONITORING_INTERVAL_MINUTES)+++def monitoring_comparison(config: MetricThresholdConfig, elapsed: timedelta) -> CheckComparison:+ if config.monitoring is None or elapsed.total_seconds() <= 0:+ raise ValueError("A Monitoring measurement needs its definition and a positive elapsed period.")+ if config.monitoring.threshold_scaling == "fixed":+ return config.comparison.model_copy(deep=True)+ factor = elapsed.total_seconds() / (config.monitoring.reference_period_minutes * 60)+ if config.comparison.operator == "between":+ assert config.comparison.bounds is not None+ return CheckComparison(+ operator="between",+ bounds=CheckThresholdBounds(+ lower=config.comparison.bounds.lower * factor, upper=config.comparison.bounds.upper * factor+ ),+ )+ assert config.comparison.value is not None+ return CheckComparison(operator=config.comparison.operator, value=config.comparison.value * factor)++class AgentCheckConfig(BaseModel):"""A check a scout run answers: re-probe the report's claim and record one verdict.@@ -387,9 +416,13 @@ def validate_metric_check_for_write(config: MetricThresholdConfig) -> None:if config.monitoring is not None:if config.baseline_value is not None and config.monitoring.baseline_period_minutes is None:raise CheckConfigValidationError("A Monitoring baseline needs its measured baseline_period_minutes.")+ source = config.query["source"]+ series = source["series"]+ if any(item.get("math") in {"weekly_active", "monthly_active"} for item in series):+ raise CheckConfigValidationError(+ "Monitoring targets cannot use rolling weekly or monthly active users. Use a distinct user count instead."+ )if config.monitoring.threshold_scaling == "linear":- source = config.query["source"]- series = source["series"]if (len(series) != 1or any((source.get("trendsFilter") or {}).get(key) for key in ("formula", "formulas", "formulaNodes"))diff --git a/products/signals/backend/report_metric_refresh.py b/products/signals/backend/report_metric_refresh.pyindex 2b4b8de2113d..fdd02c15616a 100644--- a/products/signals/backend/report_metric_refresh.py+++ b/products/signals/backend/report_metric_refresh.py@@ -31,6 +31,7 @@ from posthog.schema import ChartDisplayTypefrom posthog.clickhouse.client.connection import ClickHouseUserfrom posthog.clickhouse.query_tagging import tag_queries+from posthog.clickhouse.workload import Workloadfrom posthog.dataclasses import frozenfrom posthog.models import Team@@ -55,6 +56,10 @@ REPORT_METRIC_REFRESH_QUERY_TIMEOUT_SECONDS = 20CURRENT_REPORT_STATUSES = (SignalReport.Status.READY, SignalReport.Status.PENDING_INPUT)+class MetricQueryBudgetExhausted(TimeoutError):+ pass++@frozenclass MetricMeasurement:value: float@@ -108,6 +113,7 @@ def _run_metric_query(*,deadline: float,ch_user: ClickHouseUser = ClickHouseUser.DEFAULT,+ workload: Workload = Workload.DEFAULT,) -> tuple[dict, datetime | None]:"""Run the stored query in one derived display shape and return its first series and response.@@ -129,7 +135,7 @@ def _run_metric_query(remaining = deadline - time.monotonic()source_count = _source_count(query)if remaining < source_count:- raise TimeoutError("report metric refresh budget expired")+ raise MetricQueryBudgetExhausted("report metric refresh budget expired")timeout = (REPORT_METRIC_REFRESH_QUERY_TIMEOUT_SECONDSif math.isinf(remaining)@@ -142,6 +148,7 @@ def _run_metric_query(max_execution_time_seconds=timeout,cache_age_seconds=int(REPORT_METRIC_SNAPSHOT_FRESH_FOR.total_seconds()),ch_user=ch_user,+ workload=workload,)results = response.resultsif not isinstance(results, list) or not results or not isinstance(results[0], dict):@@ -164,9 +171,10 @@ def whole_window_value(*,deadline: float = math.inf,ch_user: ClickHouseUser = ClickHouseUser.DEFAULT,+ workload: Workload = Workload.DEFAULT,) -> tuple[float, datetime]:first_series, last_refresh = _run_metric_query(- query, team, ChartDisplayType.BOLD_NUMBER, deadline=deadline, ch_user=ch_user+ query, team, ChartDisplayType.BOLD_NUMBER, deadline=deadline, ch_user=ch_user, workload=workload)value = _finite_number(first_series.get("aggregated_value"), what="aggregate")measured_at = last_refresh if isinstance(last_refresh, datetime) else timezone.now()@@ -181,10 +189,13 @@ def longitudinal_values(*,deadline: float = math.inf,ch_user: ClickHouseUser = ClickHouseUser.DEFAULT,+ workload: Workload = Workload.DEFAULT,) -> list[float]:"""The trailing per-bucket values, oldest first, for the row-sized trend strip."""- first_series, _ = _run_metric_query(query, team, ChartDisplayType.ACTIONS_BAR, deadline=deadline, ch_user=ch_user)+ first_series, _ = _run_metric_query(+ query, team, ChartDisplayType.ACTIONS_BAR, deadline=deadline, ch_user=ch_user, workload=workload+ )raw_points = first_series.get("data")if not isinstance(raw_points, list):raise ValueError("metric query returned no buckets")@@ -198,13 +209,14 @@ def measure_metric(deadline: float,include_series: bool,ch_user: ClickHouseUser = ClickHouseUser.DEFAULT,+ workload: Workload = Workload.DEFAULT,) -> MetricMeasurement:- value, measured_at = whole_window_value(query, team, deadline=deadline, ch_user=ch_user)+ value, measured_at = whole_window_value(query, team, deadline=deadline, ch_user=ch_user, workload=workload)series: list[float] | Noneif not include_series or time.monotonic() >= deadline:return MetricMeasurement(value=value, measured_at=measured_at, series=None)try:- series = longitudinal_values(query, team, deadline=deadline, ch_user=ch_user)+ series = longitudinal_values(query, team, deadline=deadline, ch_user=ch_user, workload=workload)except Exception:# The buckets only decorate the row, so losing them must not cost the headline value.logger.exception("signals.report_metric_refresh.series_query_failed", team_id=team.id)diff --git a/products/signals/backend/scout_harness/tools/checks.py b/products/signals/backend/scout_harness/tools/checks.pyindex e96b72926d3a..d2ed69e76c9f 100644--- a/products/signals/backend/scout_harness/tools/checks.py+++ b/products/signals/backend/scout_harness/tools/checks.py@@ -36,11 +36,14 @@ from products.signals.backend.artefact_attribution import ArtefactAttributionfrom products.signals.backend.models import SignalReport, SignalReportCheck, SignalScoutRunfrom products.signals.backend.report_check_agent import AGENT_CHECK_RESULT_WINDOW, resolve_check_skill_namefrom products.signals.backend.report_check_authoring import CheckCreationError, cancel_check, create_check-from products.signals.backend.report_check_execution import CheckVerdict, record_check_verdict+from products.signals.backend.report_check_evaluation import CheckVerdict+from products.signals.backend.report_check_execution import record_check_verdictfrom products.signals.backend.report_checks import (+ MAX_CHECK_HORIZON,AgentCheckConfig,CheckInconclusiveReason,CheckOutcome,+ monitoring_window_end,parse_check_config,)from products.signals.backend.scout_harness.tools.emit import _preflight_emit_gates, _resolve_task_id@@ -77,7 +80,13 @@ def _pause_reason(check: SignalReportCheck, report_status: str, now: datetime) -These are the filters `collect_due_checks` applies besides the clock, so an undispatched checkis `due` to a run only when the coordinator would dispatch it too."""- if report_status != SignalReport.Status.RESOLVED:+ if report_status == SignalReport.Status.MONITORING:+ start = check.report.monitoring_started_at+ if check.kind != SignalReportCheck.Kind.METRIC_THRESHOLD or not check.config.get("monitoring") or start is None:+ return "its report is `monitoring`"+ if now >= min(monitoring_window_end(start, check.soak_minutes), start + MAX_CHECK_HORIZON):+ return "its Monitoring window ended"+ elif report_status != SignalReport.Status.RESOLVED:return f"its report is `{report_status}`"if check.expires_at <= now:return "its horizon passed"@@ -371,6 +380,8 @@ def list_report_checks(*, team: Team, report_id: str) -> list[ScoutCheckSummary].filter(report_id=report.id).order_by("-created_at")[:MAX_CHECKS_LISTED])+ for check in checks:+ check.report = reportrun_ids = _dispatched_run_ids(team.parent_team_id or team.id, checks)now = timezone.now()report_status = report.effective_statusdiff --git a/products/signals/backend/serializers.py b/products/signals/backend/serializers.pyindex a86467192669..650f28116e78 100644--- a/products/signals/backend/serializers.py+++ b/products/signals/backend/serializers.py@@ -2521,7 +2521,10 @@ class SignalReportArtefactSerializer(serializers.ModelSerializer):manual_adders=trusted_manual_reviewer_adders(obj, parsed, self.context.get("signals_reviewer_history")),)- if obj.type == SignalReportArtefact.ArtefactType.CHECK_RESULT and isinstance(parsed, dict):+ if obj.type in {+ SignalReportArtefact.ArtefactType.CHECK_RESULT,+ SignalReportArtefact.ArtefactType.CHECK_OBSERVATION,+ } and isinstance(parsed, dict):return self._redact_check_result(obj, parsed)if (@@ -2581,6 +2584,11 @@ class SignalReportArtefactSerializer(serializers.ModelSerializer):"observed_value": None,"baseline_value": None,"explanation": CHECK_RESULT_HIDDEN_EXPLANATION,+ **(+ {"comparison": None, "matches": None, "error": None}+ if obj.type == SignalReportArtefact.ArtefactType.CHECK_OBSERVATION+ else {}+ ),}diff --git a/products/signals/backend/temporal/agentic/scout_coordinator.py b/products/signals/backend/temporal/agentic/scout_coordinator.pyindex 5032095c5f50..2c3fd1079ee0 100644--- a/products/signals/backend/temporal/agentic/scout_coordinator.py+++ b/products/signals/backend/temporal/agentic/scout_coordinator.py@@ -181,6 +181,7 @@ class RunDueChecksOutput:# Defaulted so a workflow history written before the agent lane existed still decodes.dispatched: int = 0deferred: int = 0+ observed: int = 0@activity.defn@@ -201,6 +202,7 @@ async def run_due_signal_report_checks_activity(_input: RunDueChecksInput) -> Ruerrored=summary.errored,dispatched=summary.dispatched,deferred=summary.deferred,+ observed=summary.observed,)diff --git a/products/signals/backend/test/test_report_check_concurrency.py b/products/signals/backend/test/test_report_check_concurrency.pynew file mode 100644index 000000000000..bad60dff8fc2--- /dev/null+++ b/products/signals/backend/test/test_report_check_concurrency.py@@ -0,0 +1,143 @@+from collections.abc import Callable+from concurrent.futures import Future, ThreadPoolExecutor+from datetime import timedelta+from queue import Queue+from time import monotonic++import time_machine+from posthog.test.base import NonAtomicBaseTest+from unittest.mock import patch++from django.db import connection, transaction+from django.utils import timezone++from parameterized import parameterized++from products.signals.backend.artefact_attribution import ArtefactAttribution+from products.signals.backend.artefact_schemas import CheckObservation+from products.signals.backend.models import SignalReport, SignalReportArtefact, SignalReportCheck+from products.signals.backend.report_check_authoring import cancel_check+from products.signals.backend.report_check_execution import expire_overdue_checks, run_due_report_checks+from products.signals.backend.report_check_monitoring import MONITORING_ERROR_RETRY_AFTER, record_monitoring_observation+from products.signals.backend.report_checks import MAX_CHECK_HORIZON+from products.signals.backend.test.report_metric_test_fixtures import trends_metric_query+++@time_machine.travel("2026-10-01T12:00:00Z", tick=False)+class TestReportCheckConcurrency(NonAtomicBaseTest):+ CLASS_DATA_LEVEL_SETUP = False++ def setUp(self) -> None:+ super().setUp()+ self.worker_pids: Queue[int] = Queue()+ self.report = SignalReport.objects.create(team=self.team, status=SignalReport.Status.RESOLVED)+ now = timezone.now()+ self.check = SignalReportCheck.objects.for_team(self.team.id).create(+ team_id=self.team.id,+ report=self.report,+ title="Errors stay low",+ kind=SignalReportCheck.Kind.METRIC_THRESHOLD,+ status=SignalReportCheck.Status.ACTIVE,+ config={+ "query": trends_metric_query(series=[{"kind": "EventsNode", "event": "$pageview"}]),+ "comparison": {"operator": "lte", "value": 10},+ "monitoring": {"reference_period_minutes": 10080, "threshold_scaling": "linear"},+ },+ soak_minutes=14 * 1440,+ next_run_at=now - timedelta(hours=1),+ expires_at=now - timedelta(minutes=1),+ )++ def _start_write(self, executor: ThreadPoolExecutor, write: Callable[[], object]) -> Future[object]:+ def run() -> object:+ try:+ with connection.cursor() as cursor:+ cursor.execute("SET statement_timeout = '15s'")+ cursor.execute("SELECT pg_backend_pid()")+ self.worker_pids.put(cursor.fetchone()[0])+ return write()+ finally:+ connection.close()++ return executor.submit(run)++ def _assert_write_waits(self, future: Future[object]) -> None:+ worker_pid = self.worker_pids.get(timeout=10)+ deadline = monotonic() + 10+ with connection.cursor() as cursor:+ cursor.execute("SELECT pg_backend_pid()")+ holder_pid = cursor.fetchone()[0]+ while monotonic() < deadline:+ if future.done():+ future.result()+ self.fail("The competing write finished before the holder transaction ended")+ cursor.execute("SELECT %s = ANY(pg_blocking_pids(%s))", [holder_pid, worker_pid])+ if cursor.fetchone()[0]:+ return+ self.fail("The competing write did not wait for the holder transaction")++ @parameterized.expand([("cancelled",), ("extended",)])+ def test_expiry_sweep_preserves_a_concurrent_change(self, change: str) -> None:+ now = timezone.now()+ checks = SignalReportCheck.objects.for_team(self.team.id).filter(id=self.check.id)+ with ThreadPoolExecutor(max_workers=1) as executor:+ with transaction.atomic():+ checks.select_for_update().get()+ future = self._start_write(+ executor, lambda: expire_overdue_checks(now, disabled_monitoring_organization_ids=frozenset())+ )+ self._assert_write_waits(future)+ if change == "cancelled":+ assert cancel_check(+ self.check, reason="stopped_by_person", attribution=ArtefactAttribution.system()+ )+ else:+ checks.update(expires_at=now + timedelta(days=1), updated_at=now)+ assert future.result(timeout=10) == 0+ self.check.refresh_from_db()+ assert self.check.status == (+ SignalReportCheck.Status.CANCELLED if change == "cancelled" else SignalReportCheck.Status.ACTIVE+ )+ if change == "extended":+ assert self.check.expires_at == now + timedelta(days=1)+ assert not SignalReportArtefact.objects.filter(+ team_id=self.team.id, report=self.report, type=SignalReportArtefact.ArtefactType.CHECK_EXPIRED+ ).exists()++ @patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True)+ def test_schedule_repair_preserves_a_concurrent_observation(self, _flag: object) -> None:+ now = timezone.now()+ start = now - timedelta(days=1)+ self.report.status = SignalReport.Status.MONITORING+ self.report.monitoring_started_at = start+ self.report.save(update_fields=["status", "monitoring_started_at"])+ SignalReportCheck.objects.for_team(self.team.id).filter(id=self.check.id).update(+ measurement_start_at=start, expires_at=start + MAX_CHECK_HORIZON, updated_at=start+ )+ self.check.refresh_from_db()+ observation = CheckObservation(+ check_id=str(self.check.id),+ title=self.check.title,+ monitoring_started_at=start,+ measured_at=now,+ monitoring_ends_at=start + timedelta(minutes=self.check.soak_minutes or 0),+ error="The metric query failed.",+ explanation="The metric query failed.",+ )+ with ThreadPoolExecutor(max_workers=1) as executor:+ with transaction.atomic():+ SignalReportCheck.objects.for_team(self.team.id).select_for_update().get(id=self.check.id)+ future = self._start_write(executor, lambda: run_due_report_checks(now=now, limit=0))+ self._assert_write_waits(future)+ assert record_monitoring_observation(self.check, observation)+ future.result(timeout=10)+ self.check.refresh_from_db()+ assert self.check.last_run_at == now+ assert self.check.next_run_at == now + MONITORING_ERROR_RETRY_AFTER+ assert self.check.consecutive_errors == 1+ assert (+ SignalReportArtefact.objects.filter(+ team_id=self.team.id, report=self.report, type=SignalReportArtefact.ArtefactType.CHECK_OBSERVATION+ ).count()+ == 1+ )diff --git a/products/signals/backend/test/test_report_checks.py b/products/signals/backend/test/test_report_checks.pyindex e1bf44a999f9..ce8127e1238c 100644--- a/products/signals/backend/test/test_report_checks.py+++ b/products/signals/backend/test/test_report_checks.py@@ -4,7 +4,7 @@ from datetime import UTC, datetime, timedeltaimport time_machinefrom posthog.test.base import APIBaseTest-from unittest.mock import AsyncMock, patch+from unittest.mock import AsyncMock, MagicMock, patchfrom django.db import connectionfrom django.test import SimpleTestCase@@ -19,6 +19,7 @@ from rest_framework.test import APIRequestFactory, force_authenticatefrom temporalio.exceptions import WorkflowAlreadyStartedErrorfrom posthog.clickhouse.client.connection import ClickHouseUser+from posthog.clickhouse.workload import Workloadfrom posthog.constants import AvailableFeaturefrom posthog.models import Organization, PropertyDefinition, Teamfrom posthog.models.personal_api_key import PersonalAPIKey@@ -26,6 +27,7 @@ from posthog.models.utils import generate_random_token_personal, hash_key_valuefrom products.access_control.backend.facade.contracts import PropertyAccessLevelfrom products.access_control.backend.models.property_access_control import PropertyAccessControl+from products.product_analytics.backend.facade.contracts import TrendsQueryRunResultfrom products.signals.backend.artefact_attribution import ArtefactAttributionfrom products.signals.backend.artefact_schemas import CheckResult, Dismissalfrom products.signals.backend.enums import SignalSourceProduct, SignalSourceType@@ -53,11 +55,10 @@ from products.signals.backend.report_check_authoring import (create_checks_from_specs,replace_metric_check,)+from products.signals.backend.report_check_evaluation import CheckVerdict, evaluate_check_valuefrom products.signals.backend.report_check_execution import (CHECK_ERROR_RETRY_AFTER,- CheckVerdict,collect_due_checks,- evaluate_check_value,expire_overdue_checks,measure_check,park_report_checks_on_reopen,@@ -65,6 +66,7 @@ from products.signals.backend.report_check_execution import (resolve_check_query,run_due_report_checks,)+from products.signals.backend.report_check_monitoring import measure_monitoring_check, record_monitoring_observationfrom products.signals.backend.report_check_research import check_versionsfrom products.signals.backend.report_check_timing import metric_check_ready_at, metric_check_window_startfrom products.signals.backend.report_checks import (@@ -88,6 +90,7 @@ from products.signals.backend.report_checks import (CheckSpec,MetricThresholdConfig,authored_monitoring_interval_minutes,+ monitoring_comparison,parse_check_config,validate_metric_check_for_write,)@@ -131,6 +134,28 @@ def _threshold_config(**overrides: object) -> dict:class TestCheckComparison(SimpleTestCase):+ @parameterized.expand(+ [+ ("fixed", "fixed", {"operator": "gte", "value": 0.4}, 3, {"operator": "gte", "value": 0.4}),+ (+ "fractional_range",+ "linear",+ {"operator": "between", "bounds": {"lower": 7, "upper": 21}},+ 0.5,+ {"operator": "between", "bounds": {"lower": 0.5, "upper": 1.5}},+ ),+ ]+ )+ def test_monitoring_target_scaling_policies(+ self, _name: str, scaling: str, comparison: dict[str, object], days: float, expected: dict[str, object]+ ) -> None:+ config = MetricThresholdConfig.model_validate(+ _threshold_config(+ comparison=comparison, monitoring={"reference_period_minutes": 10080, "threshold_scaling": scaling}+ )+ )+ assert monitoring_comparison(config, timedelta(days=days)).model_dump(exclude_none=True) == expected+@parameterized.expand([("lte_under", {"operator": "lte", "value": 10}, 4.0, "passed"),@@ -234,6 +259,8 @@ class TestCheckComparison(SimpleTestCase):("average", "avg", "linear", None, False),("fixed_average", "avg", "fixed", None, True),("fixed_distinct", "dau", "fixed", None, True),+ ("rolling_weekly", "weekly_active", "fixed", None, False),+ ("rolling_monthly", "monthly_active", "fixed", None, False),("formula", "total", "linear", {"formulas": ["A * 2"]}, False),("missing_baseline_period", "total", "fixed", None, False),]@@ -584,6 +611,406 @@ class TestReportCheckExecution(APIBaseTest):fields.update(overrides)return SignalReportCheck.objects.for_team(self.team.id).create(**fields)+ def _monitoring_check(self, *, start: datetime, **overrides: object) -> SignalReportCheck:+ SignalReport.objects.filter(id=self.report.id, team_id=self.team.id).update(+ status=SignalReport.Status.MONITORING, monitoring_started_at=start+ )+ self.report.refresh_from_db()+ query = trends_metric_query(series=[{"kind": "EventsNode", "event": "$pageview"}], date_from="-7d")+ query["source"]["dateRange"].update(daysOfWeek=[1, 2, 3, 4, 5], excludeIncompletePeriods=True)+ fields = {+ "config": _threshold_config(+ query=query,+ comparison={"operator": "lte", "value": 70},+ monitoring={"reference_period_minutes": 10080, "threshold_scaling": "linear"},+ ),+ "measurement_start_at": start,+ "next_run_at": start + timedelta(days=14),+ "expires_at": start + timedelta(days=90),+ "soak_minutes": 14 * 1440,+ "runs_remaining": 1,+ }+ fields.update(overrides)+ return self._check(**fields)++ @parameterized.expand([("daily", None, 1440), ("authored", 360, 360)])+ def test_monitoring_samples_repeat_without_consuming_final_runs(+ self, _name: str, interval: int | None, minutes: int+ ) -> None:+ start = datetime(2026, 10, 1, 12, 25, 17, tzinfo=UTC)+ with (+ time_machine.travel(start, tick=False),+ patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True),+ ):+ check = self._monitoring_check(start=start, run_interval_minutes=interval)+ with patch(+ "products.signals.backend.report_check_monitoring.measure_metric",+ return_value=MetricMeasurement(value=8, measured_at=start, series=None),+ ) as measure:+ run_due_report_checks(now=start + timedelta(minutes=minutes - 1))+ measure.assert_not_called()+ cutoff = start + timedelta(days=1)+ run_due_report_checks(now=cutoff)+ measured_query = measure.call_args.args[0]+ assert measured_query["source"]["dateRange"] == {+ "date_from": start.isoformat(),+ "date_to": cutoff.isoformat(),+ "explicitDate": True,+ "excludeIncompletePeriods": False,+ "daysOfWeek": [1, 2, 3, 4, 5],+ }+ assert measured_query["source"]["series"][0]["event"] == "$pageview"+ with patch(+ "products.signals.backend.report_check_monitoring.measure_metric",+ return_value=MetricMeasurement(value=35, measured_at=start, series=None),+ ):+ run_due_report_checks(now=start + timedelta(days=3))+ check.refresh_from_db()+ self.report.refresh_from_db()+ observations = [json.loads(row.content) for row in self.report.artefacts.filter(type="check_observation")]+ assert sorted((row["observed_value"], row["comparison"]["value"], row["matches"]) for row in observations) == [+ (8, 10, True),+ (35, 30, False),+ ]+ assert check.status == SignalReportCheck.Status.ACTIVE+ assert check.runs_remaining == 1 and check.last_outcome is None+ assert check.next_run_at == start + timedelta(days=3, minutes=minutes)+ assert self.report.status == SignalReport.Status.MONITORING+ assert self._results() == []+ assert SignalReport.objects.filter(team_id=self.team.id).count() == 1++ @parameterized.expand([("sampled", 720), ("shorter_than_cadence", 180), ("no_soak", None)])+ @time_machine.travel("2026-10-02T12:00:00Z", tick=False)+ def test_interim_samples_stop_at_the_soak_deadline_without_a_final_outcome(+ self, _name: str, soak: int | None+ ) -> None:+ start = timezone.now()+ end = start + timedelta(minutes=soak or 0)+ check = self._monitoring_check(start=start, soak_minutes=soak, run_interval_minutes=360)+ with (+ patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True),+ patch(+ "products.signals.backend.report_check_monitoring.measure_metric",+ return_value=MetricMeasurement(value=8, measured_at=start, series=None),+ ) as measure,+ ):+ if soak == 720:+ run_due_report_checks(now=end - timedelta(minutes=1))+ content = json.loads(self.report.artefacts.get(type="check_observation").content)+ assert datetime.fromisoformat(content["monitoring_ends_at"]) == end+ assert content["observed_value"] == 8+ run_due_report_checks(now=end)+ run_due_report_checks(now=end + timedelta(days=1))+ assert measure.call_count == (1 if soak == 720 else 0)+ check.refresh_from_db()+ self.report.refresh_from_db()+ assert check.next_run_at == end+ assert check.status == SignalReportCheck.Status.ACTIVE+ assert check.last_outcome is None and check.runs_remaining == 1+ assert self.report.status == SignalReport.Status.MONITORING+ assert self._results() == []++ @time_machine.travel("2026-10-02T12:00:00Z", tick=False)+ def test_tick_budget_exhaustion_defers_without_spending_the_query_error_budget(self) -> None:+ now = timezone.now()+ query = trends_metric_query(+ series=[+ {"kind": "EventsNode", "event": "checkout_failed"},+ {"kind": "EventsNode", "event": "checkout_started"},+ ]+ )+ query["source"]["trendsFilter"] = {"formula": "A / B"}+ check = self._monitoring_check(+ start=now - timedelta(days=1),+ next_run_at=now,+ consecutive_errors=MAX_CONSECUTIVE_CHECK_ERRORS - 1,+ config=_threshold_config(+ query=query, monitoring={"reference_period_minutes": 10080, "threshold_scaling": "fixed"}+ ),+ )+ with (+ patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True),+ patch("time.monotonic", return_value=100.0),+ patch(+ "products.product_analytics.backend.hogql_queries.trends.trends_query_runner.TrendsQueryRunner",+ ) as runner,+ ):+ run_query = runner.return_value.run+ run_query.return_value = TrendsQueryRunResult(results=[{"aggregated_value": 2.0}], last_refresh=now)+ with patch("products.signals.backend.report_check_execution.CHECK_RUN_TIME_BUDGET_SECONDS", 1):+ for _ in range(3):+ run_due_report_checks(now=now)+ run_query.assert_not_called()+ check.refresh_from_db()+ assert check.status == SignalReportCheck.Status.ACTIVE+ assert check.consecutive_errors == MAX_CONSECUTIVE_CHECK_ERRORS - 1+ assert check.last_run_at is None and check.next_run_at == now+ assert not self.report.artefacts.filter(type="check_observation").exists()+ run_due_report_checks(now=now)+ run_query.assert_called_once()+ assert runner.call_args.kwargs["workload"] == Workload.OFFLINE+ assert runner.call_args.kwargs["ch_user"] == ClickHouseUser.SIGNALS+ check.refresh_from_db()+ assert check.consecutive_errors == 0+ content = json.loads(self.report.artefacts.get(type="check_observation").content)+ assert content["observed_value"] == 2 and content["error"] is None++ @parameterized.expand(+ [+ ("duplicate",),+ ("cancelled",),+ ("new_period",),+ ("resolved",),+ ("flag_off",),+ ("config_changed",),+ ("soak_elapsed",),+ ("soak_changed",),+ ]+ )+ @time_machine.travel("2026-10-02T12:00:00Z", tick=False)+ def test_stale_monitoring_observations_are_not_saved(self, race: str) -> None:+ start = timezone.now() - timedelta(days=1)+ check = self._monitoring_check(start=start, next_run_at=timezone.now() - timedelta(minutes=1))+ with patch(+ "products.signals.backend.report_check_monitoring.measure_metric",+ return_value=MetricMeasurement(value=8, measured_at=timezone.now(), series=None),+ ):+ observation = measure_monitoring_check(check, now=timezone.now(), deadline=time.monotonic() + 60)+ if race == "cancelled":+ SignalReportCheck.objects.for_team(self.team.id).filter(id=check.id).update(+ status=SignalReportCheck.Status.CANCELLED+ )+ elif race == "new_period":+ SignalReport.objects.filter(id=self.report.id, team_id=self.team.id).update(+ monitoring_started_at=timezone.now()+ )+ elif race == "resolved":+ SignalReport.objects.filter(id=self.report.id, team_id=self.team.id).update(+ status=SignalReport.Status.RESOLVED+ )+ elif race in {"soak_elapsed", "soak_changed"}:+ SignalReportCheck.objects.for_team(self.team.id).filter(id=check.id).update(+ soak_minutes=1440 if race == "soak_elapsed" else 2880+ )+ elif race == "config_changed":+ SignalReportCheck.objects.for_team(self.team.id).filter(id=check.id).update(config=_threshold_config())+ with patch(+ "products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=race != "flag_off"+ ):+ if race == "duplicate":+ record_monitoring_observation(check, observation)+ record_monitoring_observation(check, observation)+ assert self.report.artefacts.filter(type="check_observation").count() == (1 if race == "duplicate" else 0)++ @parameterized.expand(+ [+ ("due", True, "metric", -1, "due"),+ ("scheduled", True, "metric", 60, "scheduled"),+ ("window_ended", True, "window_ended", -1, "paused"),+ ("expired", True, "expired", -1, "paused"),+ ("missing_start", True, "missing_start", -1, "paused"),+ ("agent", True, "agent", -1, "paused"),+ ("legacy_metric", True, "legacy_metric", -1, "paused"),+ ("flag_off", False, "metric", -1, "due"),+ ]+ )+ def test_scout_monitoring_run_state_matches_coordinator(+ self, _name: str, flag: bool, variant: str, due_in_minutes: int, expected: str+ ) -> None:+ now = timezone.now()+ with (+ time_machine.travel(now, tick=False),+ patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=flag),+ ):+ start = now - timedelta(days=1)+ check = self._monitoring_check(start=start, next_run_at=now + timedelta(minutes=due_in_minutes))+ if variant == "agent":+ check.kind = SignalReportCheck.Kind.AGENT+ check.config = {"instructions": "Investigate the report."}+ elif variant == "legacy_metric":+ check.config.pop("monitoring")+ elif variant == "expired":+ check.expires_at = now+ elif variant == "window_ended":+ check.soak_minutes = 1440+ elif variant == "missing_start":+ SignalReport.objects.filter(team_id=self.team.id, id=self.report.id).update(monitoring_started_at=None)+ check.save()+ (listed,) = list_report_checks(team=self.team, report_id=str(self.report.id))+ assert listed.run_state == expected+ assert {row.id for row in collect_due_checks(now)} == ({check.id} if expected == "due" else set())++ def test_flag_off_restores_the_legacy_soak_and_whole_window(self) -> None:+ start = datetime(2026, 10, 1, 12, tzinfo=UTC)+ with time_machine.travel(start, tick=False):+ check = self._monitoring_check(start=start)+ with (+ patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True),+ patch(+ "products.signals.backend.report_check_monitoring.measure_metric",+ return_value=MetricMeasurement(value=8, measured_at=start, series=None),+ ),+ ):+ run_due_report_checks(now=start + timedelta(days=1))+ with (+ patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=False),+ patch(_MEASURE, return_value=MetricMeasurement(value=2, measured_at=start, series=None)) as measure,+ ):+ run_due_report_checks(now=start + timedelta(days=2))+ measure.assert_not_called()+ check.refresh_from_db()+ assert check.next_run_at == start + timedelta(days=14)+ with time_machine.travel(start + timedelta(days=14), tick=False):+ run_due_report_checks()+ check.refresh_from_db()+ assert check.status == SignalReportCheck.Status.PASSED+ assert check.runs_remaining == 0++ @parameterized.expand([("success", False), ("query_error", True)])+ def test_flag_recovery_restores_the_latest_monitoring_attempt_schedule(self, _name: str, query_error: bool) -> None:+ start = datetime(2026, 10, 1, 12, tzinfo=UTC)+ sample_at = start + timedelta(days=1)+ interval = CHECK_ERROR_RETRY_AFTER if query_error else timedelta(days=1)+ with time_machine.travel(sample_at, tick=False):+ check = self._monitoring_check(start=start)+ with (+ patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True),+ patch(+ "products.signals.backend.report_check_monitoring.measure_metric",+ side_effect=RuntimeError("Query unavailable") if query_error else None,+ return_value=MetricMeasurement(value=8, measured_at=sample_at, series=None),+ ),+ ):+ run_due_report_checks()+ with patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=False):+ run_due_report_checks(now=sample_at + interval)+ check.refresh_from_db()+ assert check.next_run_at == start + timedelta(days=14)+ with (+ patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True),+ patch(+ "products.signals.backend.report_check_monitoring.measure_metric",+ return_value=MetricMeasurement(value=9, measured_at=sample_at, series=None),+ ) as measure,+ ):+ run_due_report_checks(now=sample_at + interval - timedelta(minutes=1))+ measure.assert_not_called()+ check.refresh_from_db()+ assert check.next_run_at == sample_at + interval+ run_due_report_checks(now=sample_at + interval)+ measure.assert_called_once()+ assert self.report.artefacts.filter(type="check_observation").count() == 2++ @parameterized.expand([("early", 3, 41, False), ("late", 80, 91, False), ("repair_backlog", 80, 91, True)])+ def test_flag_recovery_restores_the_original_monitoring_horizon(+ self, _name: str, armed_day: int, recovery_day: int, backlog: bool+ ) -> None:+ start = datetime(2026, 10, 1, 12, tzinfo=UTC)+ with time_machine.travel(start + timedelta(days=armed_day), tick=False):+ check = self._monitoring_check(start=start, status=SignalReportCheck.Status.PENDING)+ with patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=False):+ run_due_report_checks()+ recovery_at = start + timedelta(days=recovery_day)+ with (+ time_machine.travel(recovery_at, tick=False),+ patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True),+ patch(+ "products.signals.backend.report_check_monitoring.measure_metric",+ return_value=MetricMeasurement(value=8, measured_at=recovery_at, series=None),+ ) as measure,+ ):+ with patch(+ "products.signals.backend.report_check_execution.MAX_CHECK_PARKS_PER_TICK", 0 if backlog else 500+ ):+ run_due_report_checks()+ check.refresh_from_db()+ if not backlog:+ assert check.expires_at == start + MAX_CHECK_HORIZON+ if recovery_day >= 90:+ measure.assert_not_called()+ assert check.status == SignalReportCheck.Status.EXPIRED+ assert not self.report.artefacts.filter(type="check_observation").exists()+ else:+ measure.assert_not_called()+ assert check.status == SignalReportCheck.Status.ACTIVE+ assert not self.report.artefacts.filter(type="check_observation").exists()++ def test_reenabling_monitoring_after_a_legacy_error_does_not_sample_past_the_soak(self) -> None:+ now = datetime(2026, 10, 1, 12, tzinfo=UTC)+ start = now - timedelta(days=3)+ with time_machine.travel(now, tick=False):+ check = self._monitoring_check(start=start, status=SignalReportCheck.Status.PENDING)+ with patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=False):+ run_due_report_checks()+ check.refresh_from_db()+ due_at = check.next_run_at+ with (+ time_machine.travel(due_at, tick=False),+ patch(_MEASURE, side_effect=RuntimeError("Query unavailable")),+ ):+ run_due_report_checks()+ sample_at = due_at + CHECK_ERROR_RETRY_AFTER+ with (+ time_machine.travel(sample_at, tick=False),+ patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True),+ patch(+ "products.signals.backend.report_check_monitoring.measure_metric",+ return_value=MetricMeasurement(value=8, measured_at=sample_at, series=None),+ ) as measure,+ ):+ run_due_report_checks()+ run_due_report_checks(now=sample_at + timedelta(minutes=1))+ measure.assert_not_called()+ check.refresh_from_db()+ assert check.measurement_start_at == start+ assert check.next_run_at == start + timedelta(days=14)+ assert not self.report.artefacts.filter(type="check_observation").exists()++ @parameterized.expand(+ [+ ("retry", 0, SignalReportCheck.Status.ACTIVE),+ ("error_limit", MAX_CONSECUTIVE_CHECK_ERRORS - 1, SignalReportCheck.Status.ERRORED),+ ("rolling_active_users", 0, SignalReportCheck.Status.ACTIVE, "weekly_active"),+ ("query_timeout", 0, SignalReportCheck.Status.ACTIVE, None, TimeoutError),+ ]+ )+ def test_monitoring_query_errors_remain_operational_errors(+ self,+ _name: str,+ errors: int,+ expected: str,+ math: str | None = None,+ query_error: type[Exception] = RuntimeError,+ ) -> None:+ now = timezone.now()+ check = self._monitoring_check(start=now - timedelta(days=1), consecutive_errors=errors)+ if math is not None:+ check.config["query"]["source"]["series"][0]["math"] = math+ check.config["monitoring"]["threshold_scaling"] = "fixed"+ check.save(update_fields=["config"])+ with (+ patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True),+ patch(+ "products.signals.backend.report_check_monitoring.measure_metric",+ side_effect=query_error(+ "metric query timed out" if query_error is TimeoutError else "private query detail"+ ),+ ) as measure,+ ):+ run_due_report_checks(now=now)+ if math is not None:+ measure.assert_not_called()+ check.refresh_from_db()+ content = json.loads(self.report.artefacts.get(type="check_observation").content)+ assert "private query detail" not in content["error"]+ assert content["observed_value"] is None and content["matches"] is None+ assert check.status == expected and check.runs_remaining == 1+ assert check.last_outcome is None+ assert check.consecutive_errors == errors + 1+ assert check.next_run_at == now + CHECK_ERROR_RETRY_AFTER+ self.report.refresh_from_db()+ assert self.report.status == SignalReport.Status.MONITORING+def _results(self) -> list[SignalReportArtefact]:return list(SignalReportArtefact.objects.filter(@@ -1535,7 +1962,11 @@ class TestReportCheckAPI(APIBaseTest):result = SignalReportArtefact.objects.get(report=report, type=SignalReportArtefact.ArtefactType.CHECK_RESULT)assert result.team_id == child.id- def test_a_property_restricted_member_cannot_read_a_checks_query_or_measured_values(self) -> None:+ @parameterized.expand([("check_result",), ("check_observation",)])+ @patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True)+ def test_a_property_restricted_member_cannot_read_a_checks_query_or_measured_values(+ self, artefact_type: str, _flag: MagicMock+ ) -> None:secret_pageviews = trends_metric_query(series=[{@@ -1547,16 +1978,30 @@ class TestReportCheckAPI(APIBaseTest):}])- check_id = str(- self._create(- title="Enterprise pageviews stay low",- config=_threshold_config(query=secret_pageviews, baseline_value=3),- ).id- )+ config = _threshold_config(query=secret_pageviews, baseline_value=3)+ if artefact_type == "check_observation":+ config["monitoring"] = {+ "reference_period_minutes": 10080,+ "threshold_scaling": "linear",+ "baseline_period_minutes": 10080,+ }+ check_id = str(self._create(title="Enterprise pageviews stay low", config=config).id)+ if artefact_type == "check_observation":+ start = timezone.now() - timedelta(days=2)+ SignalReportCheck.objects.for_team(self.team.id).filter(id=check_id).update(soak_minutes=3 * 1440)+ SignalReport.objects.filter(id=self.report.id, team_id=self.team.id).update(+ status=SignalReport.Status.MONITORING, monitoring_started_at=start+ )+SignalReportCheck.objects.for_team(self.team.id).filter(id=check_id).update(next_run_at=timezone.now() - timedelta(minutes=1), measurement_start_at=timezone.now() - timedelta(days=32))- with patch(_MEASURE, return_value=MetricMeasurement(value=4.0, measured_at=timezone.now(), series=None)):+ with patch(+ _MEASURE+ if artefact_type == "check_result"+ else "products.signals.backend.report_check_monitoring.measure_metric",+ return_value=MetricMeasurement(value=4.0, measured_at=timezone.now(), series=None),+ ):run_due_report_checks()artefacts_url = f"/api/projects/{self.team.id}/signals/reports/{self.report.id}/artefacts/"@@ -1564,7 +2009,7 @@ class TestReportCheckAPI(APIBaseTest):return next(row["content"]for row in self.client.get(artefacts_url).json()["results"]- if row["type"] == "check_result"+ if row["type"] == artefact_type)assert check_result()["observed_value"] == 4.0@@ -1588,7 +2033,12 @@ class TestReportCheckAPI(APIBaseTest):assert config["comparison"] == {"operator": "lte", "value": 10}hidden = check_result()- assert hidden["outcome"] == "passed"+ if artefact_type == "check_result":+ assert hidden["outcome"] == "passed"+ else:+ assert hidden["comparison"] is None+ assert hidden["matches"] is None+ assert hidden["error"] is Noneassert hidden["observed_value"] is Noneassert hidden["baseline_value"] is Noneassert hidden["explanation"] == CHECK_RESULT_HIDDEN_EXPLANATION@@ -2355,21 +2805,105 @@ class TestPendingChecks(APIBaseTest):assert checks[0].next_run_at == expected_dueassert collect_due_checks(expected_due + timedelta(minutes=1)) == [checks[0]]+ @parameterized.expand([("successful", 0, False, 24), ("first_error", 0, True, 6), ("second_error", 1, True, 6)])+ def test_monitoring_definition_arms_and_restores_its_observation_cadence(+ self, _name: str, prior_errors: int, query_error: bool, delay_hours: int+ ) -> None:+ now = timezone.now()+ with (+ time_machine.travel(now, tick=False),+ patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True),+ ):+ check = create_check(+ report=self.report,+ title="Errors stay low",+ kind=SignalReportCheck.Kind.METRIC_THRESHOLD,+ config=_threshold_config(+ query=trends_metric_query(series=[{"kind": "EventsNode", "event": "$pageview"}], date_from="-120d"),+ monitoring={"reference_period_minutes": 10080, "threshold_scaling": "linear"},+ ),+ soak_minutes=14 * 1440,+ attribution=ArtefactAttribution.system(),+ )+ with self.captureOnCommitCallbacks(execute=True):+ self.report.save(+ update_fields=self.report.transition_to(SignalReport.Status.MONITORING, monitoring_started_at=now)+ )+ check.refresh_from_db()+ assert check.status == SignalReportCheck.Status.ACTIVE+ assert check.next_run_at == now + timedelta(days=1)+ assert check.measurement_start_at == now+ assert check.expires_at == now + MAX_CHECK_HORIZON+ if prior_errors:+ check.consecutive_errors = prior_errors+ check.save(update_fields=["consecutive_errors"])+ sample_at = now + timedelta(days=1)+ with (+ time_machine.travel(sample_at, tick=False),+ patch(+ "products.signals.backend.report_check_monitoring.measure_metric",+ return_value=MetricMeasurement(value=8, measured_at=sample_at, series=None),+ side_effect=RuntimeError("query failed") if query_error else None,+ ) as measure,+ ):+ run_due_report_checks()+ with time_machine.travel(sample_at + timedelta(minutes=5), tick=False):+ with self.captureOnCommitCallbacks(execute=True):+ self.report.save(update_fields=self.report.transition_to(SignalReport.Status.SUPPRESSED))+ self.report.save(update_fields=self.report.transition_to(self.report.restore_target_status()))+ run_due_report_checks()+ measure.assert_called_once()+ check.refresh_from_db()+ retry_at = sample_at + timedelta(hours=delay_hours)+ assert check.measurement_start_at == now+ assert check.next_run_at == retry_at+ assert check.consecutive_errors == (prior_errors + 1 if query_error else 0)+ with time_machine.travel(retry_at - timedelta(minutes=1), tick=False):+ run_due_report_checks()+ measure.assert_called_once()+ measure.side_effect = None+ with time_machine.travel(retry_at, tick=False):+ run_due_report_checks()+ assert measure.call_count == 2+ check.refresh_from_db()+ assert check.consecutive_errors == 0+ assert self.report.artefacts.filter(type="check_observation").count() == 2+def test_a_delayed_monitoring_callback_cannot_arm_a_newer_period(self) -> None:- check = self._pending()now = timezone.now()old_start = now - timedelta(days=3)new_start = now - timedelta(hours=1)with (time_machine.travel(now, tick=False),patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True),+ patch(+ "products.signals.backend.report_check_monitoring.measure_metric",+ side_effect=RuntimeError("query failed"),+ ),):- with self.captureOnCommitCallbacks(execute=False) as old_callbacks:+ check = create_check(+ report=self.report,+ title="Errors stay low",+ kind=SignalReportCheck.Kind.METRIC_THRESHOLD,+ config=_threshold_config(monitoring={"reference_period_minutes": 10080, "threshold_scaling": "linear"}),+ soak_minutes=7 * 1440,+ attribution=ArtefactAttribution.system(),+ )+ with self.captureOnCommitCallbacks(execute=True):self.report.save(update_fields=self.report.transition_to(SignalReport.Status.MONITORING, monitoring_started_at=old_start))+ check.consecutive_errors = MAX_CONSECUTIVE_CHECK_ERRORS - 2+ check.save(update_fields=["consecutive_errors"])+ with time_machine.travel(old_start + timedelta(days=1), tick=False):+ run_due_report_checks()+ check.refresh_from_db()+ assert check.consecutive_errors == MAX_CONSECUTIVE_CHECK_ERRORS - 1+ with self.captureOnCommitCallbacks(execute=False) as old_callbacks:+ self.report.save(update_fields=self.report.transition_to(SignalReport.Status.SUPPRESSED))+ self.report.save(update_fields=self.report.transition_to(self.report.restore_target_status()))self.report.save(update_fields=self.report.transition_to(SignalReport.Status.READY))with self.captureOnCommitCallbacks(execute=False) as new_callbacks:self.report.save(@@ -2387,6 +2921,12 @@ class TestPendingChecks(APIBaseTest):check.refresh_from_db()assert check.status == SignalReportCheck.Status.ACTIVEassert check.measurement_start_at == new_start+ assert check.next_run_at == new_start + timedelta(days=1)+ with time_machine.travel(check.next_run_at, tick=False):+ run_due_report_checks()+ check.refresh_from_db()+ assert check.status == SignalReportCheck.Status.ACTIVE+ assert check.consecutive_errors == 1def test_restoring_monitoring_rearms_checks_from_the_original_period(self) -> None:check = self._pending()diff --git a/products/signals/backend/test/test_report_metric_refresh.py b/products/signals/backend/test/test_report_metric_refresh.pyindex ba171b96d917..cb4f67d8eb56 100644--- a/products/signals/backend/test/test_report_metric_refresh.py+++ b/products/signals/backend/test/test_report_metric_refresh.py@@ -12,6 +12,7 @@ from rest_framework import statusfrom posthog.schema import ChartDisplayTypefrom posthog.clickhouse.client.connection import ClickHouseUser+from posthog.clickhouse.workload import Workloadfrom posthog.constants import AvailableFeaturefrom products.access_control.backend.facade.contracts import PropertyAccessLevel@@ -348,6 +349,7 @@ class TestReportMetricQueryShapes(APIBaseTest):assert run_query.call_args.kwargs["cache_age_seconds"] == 15 * 60assert run_query.call_args.kwargs["ch_user"] == (ch_user or ClickHouseUser.DEFAULT)+ assert run_query.call_args.kwargs["workload"] == Workload.DEFAULTdef test_skips_the_strip_when_the_deadline_passes_after_the_headline(self) -> None:measured_at = timezone.now()diff --git a/products/signals/backend/test/test_signal_report_artefact_api.py b/products/signals/backend/test/test_signal_report_artefact_api.pyindex e2f38307b6cd..b8ef6bb878b8 100644--- a/products/signals/backend/test/test_signal_report_artefact_api.py+++ b/products/signals/backend/test/test_signal_report_artefact_api.py@@ -142,6 +142,25 @@ class TestSignalReportArtefactViewSet(APIBaseTest):types = {row["type"] for row in data["results"]}assert types == {"suggested_reviewers", "priority_judgment"}+ @parameterized.expand([("enabled", True), ("disabled", False)])+ def test_monitoring_observation_reads_and_counts_follow_the_flag(self, _name: str, enabled: bool) -> None:+ report = self._create_report()+ observation = self._create_artefact(+ report,+ artefact_type="check_observation",+ content={"check_id": str(uuid.uuid4()), "explanation": "Interim measurement."},+ )+ with patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=enabled):+ response = self.client.get(self._list_url(str(report.id)))+ assert response.status_code == status.HTTP_200_OK+ assert response.json()["count"] == int(enabled)+ assert self.client.get(self._detail_url(str(report.id), str(observation.id))).status_code == (+ status.HTTP_200_OK if enabled else status.HTTP_404_NOT_FOUND+ )+ report_url = f"/api/projects/{self.team.id}/signals/reports/"+ assert self.client.get(f"{report_url}{report.id}/").json()["artefact_count"] == int(enabled)+ assert self.client.get(report_url).json()["results"][0]["artefact_count"] == int(enabled)+def test_list_enriches_suggested_reviewers_with_user(self):member = self._create_org_member("[email protected]", github_login="Alice")report = self._create_report()diff --git a/products/signals/backend/views.py b/products/signals/backend/views.pyindex 373b2a24d19c..edf18f3a7abf 100644--- a/products/signals/backend/views.py+++ b/products/signals/backend/views.py@@ -1378,7 +1378,14 @@ class SignalReportViewSet(# so the main query doesn't LEFT JOIN + GROUP BY the full artefact tableartefact_count_subquery = Subquery(SignalReportArtefact.objects.filter(report_id=OuterRef("id"))- .exclude(type__in=SignalReportArtefact.SYSTEM_SCORING_ARTEFACT_TYPES)+ .exclude(+ type__in=SignalReportArtefact.SYSTEM_SCORING_ARTEFACT_TYPES+ | (+ {SignalReportArtefact.ArtefactType.CHECK_OBSERVATION}+ if not self._report_monitoring_enabled+ else set()+ )+ ).values("report_id").annotate(count=Count("*")).values("count"),@@ -2695,7 +2702,9 @@ class SignalReportViewSet(# page, rather than a correlated subquery the sort makes Postgres run for every report in# the team. The serializer reads both off the instance, as it does for the other actions.with tracer.start_as_current_span("signals.reports.list.fetch_artefact_fields"):- artefact_counts = SignalReportArtefact.counts_by_report(report_ids)+ artefact_counts = SignalReportArtefact.counts_by_report(+ report_ids, monitoring_enabled=self._report_monitoring_enabled+ )live_channel_ids = SignalReportArtefact.live_channel_ids_by_report(report_ids)for report in reports:report.__dict__.update(@@ -5567,6 +5576,8 @@ class SignalReportArtefactViewSet(report_id=self._validated_report_id(),team=self.team,).exclude(report__status=SignalReport.Status.DELETED)+ if not organization_report_monitoring_enabled(self.organization.id):+ queryset = queryset.exclude(type=SignalReportArtefact.ArtefactType.CHECK_OBSERVATION)# Scoring rows are staff-only on every artefact route, not only in the log.if not self.request.user.is_staff:queryset = queryset.exclude(type__in=SignalReportArtefact.SYSTEM_SCORING_ARTEFACT_TYPES)diff --git a/products/signals/frontend/generated/api.schemas.ts b/products/signals/frontend/generated/api.schemas.tsindex ec6b801ada52..bda495608c62 100644--- a/products/signals/frontend/generated/api.schemas.ts+++ b/products/signals/frontend/generated/api.schemas.ts@@ -3258,6 +3258,7 @@ export interface SignalReportStateRequestApi {* * `work_release` - Work Release* * `pull_request` - Pull Request* * `check_result` - Check Result+ * * `check_observation` - Check Observation* * `check_scheduled` - Check Scheduled* * `check_expired` - Check Expired* * `check_cancelled` - Check Cancelled@@ -3296,6 +3297,7 @@ export const SignalReportArtefactArtefactTypeEnumApi = {WorkRelease: 'work_release',PullRequest: 'pull_request',CheckResult: 'check_result',+ CheckObservation: 'check_observation',CheckScheduled: 'check_scheduled',CheckExpired: 'check_expired',CheckCancelled: 'check_cancelled',diff --git a/products/signals/frontend/generated/api.ts b/products/signals/frontend/generated/api.tsindex 6ff419d8eceb..dae271fb9ea9 100644--- a/products/signals/frontend/generated/api.ts+++ b/products/signals/frontend/generated/api.ts@@ -1157,7 +1157,7 @@ export const getSignalsReportArtefactsDestroyUrl = (projectId: string, reportId:}/**- * Delete an artefact, addressed by id. Deleting the latest row of a status type reverts the report's canonical status to the previous version (latest-wins over what remains). `task_run` artefacts are an append-only work log and cannot be deleted. Neither can the types this API cannot write, which the pipeline owns: `autostart_skip`, `check_cancelled`, `check_expired`, `check_result`, `check_scheduled`, `code_review`, `impact_measurement_plan`, `implementation_decision`, `implementation_dispatch`, `implementation_handover`, `implementation_replacement`, `pull_request`, `ranking_score`, `report_link`, `summary_change`, `task_run`, `title_change`, `video_segment`, `work_claim`, `work_release`.+ * Delete an artefact, addressed by id. Deleting the latest row of a status type reverts the report's canonical status to the previous version (latest-wins over what remains). `task_run` artefacts are an append-only work log and cannot be deleted. Neither can the types this API cannot write, which the pipeline owns: `autostart_skip`, `check_cancelled`, `check_expired`, `check_observation`, `check_result`, `check_scheduled`, `code_review`, `impact_measurement_plan`, `implementation_decision`, `implementation_dispatch`, `implementation_handover`, `implementation_replacement`, `pull_request`, `ranking_score`, `report_link`, `summary_change`, `task_run`, `title_change`, `video_segment`, `work_claim`, `work_release`.* @summary Delete an artefact*/export const signalsReportArtefactsDestroy = async (diff --git a/products/signals/frontend/inbox/components/detail/ArtefactLogList.stories.tsx b/products/signals/frontend/inbox/components/detail/ArtefactLogList.stories.tsxindex c28a28bc2987..1bc7d4e2f1e7 100644--- a/products/signals/frontend/inbox/components/detail/ArtefactLogList.stories.tsx+++ b/products/signals/frontend/inbox/components/detail/ArtefactLogList.stories.tsx@@ -371,3 +371,55 @@ export const RankingScoreNarrow: Story = {),],}++export const MonitoringObservations: Story = {+ parameters: { mockDate: '2026-10-04T12:00:00Z', featureFlags: ['signals-report-monitoring'] },+ args: {+ artefacts: [+ {+ id: 'observation-two',+ type: 'check_observation',+ created_at: '2026-10-04T12:00:00Z',+ content: {+ check_id: 'example-check',+ title: 'Checkout errors stay low',+ monitoring_started_at: '2026-10-01T12:00:00Z',+ measured_at: '2026-10-04T12:00:00Z',+ monitoring_ends_at: '2026-10-05T00:00:00Z',+ unit: 'errors',+ value_format: 'number',+ observed_value: 35,+ comparison: { operator: 'lte', value: 30 },+ matches: false,+ explanation:+ 'Interim measurement: Checkout errors stay low (35.0) is more than upper threshold (30.0)',+ },+ },+ {+ id: 'observation-one',+ type: 'check_observation',+ created_at: '2026-10-02T12:00:00Z',+ content: {+ check_id: 'example-check',+ title: 'Checkout errors stay low',+ monitoring_started_at: '2026-10-01T12:00:00Z',+ measured_at: '2026-10-02T12:00:00Z',+ monitoring_ends_at: '2026-10-05T00:00:00Z',+ unit: 'errors',+ value_format: 'number',+ observed_value: 8,+ comparison: { operator: 'lte', value: 10 },+ matches: true,+ explanation: 'Interim measurement: Checkout errors stay low (8.0) is at most 10.0, as expected',+ },+ },+ ],+ },+ decorators: [+ (Story) => (+ <div className="max-w-[32rem]">+ <Story />+ </div>+ ),+ ],+}diff --git a/products/signals/frontend/inbox/components/detail/ArtefactLogList.test.tsx b/products/signals/frontend/inbox/components/detail/ArtefactLogList.test.tsxindex 756f44b701a4..1eeb1c952e1a 100644--- a/products/signals/frontend/inbox/components/detail/ArtefactLogList.test.tsx+++ b/products/signals/frontend/inbox/components/detail/ArtefactLogList.test.tsx@@ -56,6 +56,54 @@ describe('ArtefactLogList', () => {expect(screen.getByText('The measurement window cannot fit before expiry.')).toBeInTheDocument()})+ it.each([+ ['on track', 8, { operator: 'lte', value: 30 }, 'number', 'at most 30'],+ ['not on track', 35, { operator: 'lte', value: 30 }, 'number', 'at most 30'],+ ['not on track', 10, { operator: 'gte', value: 10.00011574 }, 'count', 'at least 10.00011574'],+ ['not on track', 10, { operator: 'lte', value: 9.99988426 }, 'count', 'at most 9.99988426'],+ [+ 'not on track',+ 10,+ { operator: 'between', bounds: { lower: 10.00011574, upper: 20.00011574 } },+ 'count',+ 'between 10.00011574 errors and 20.00011574',+ ],+ [+ 'not on track',+ 10,+ { operator: 'between', bounds: { lower: 0, upper: 9.99988426 } },+ 'count',+ 'between 0 errors and 9.99988426',+ ],+ ])('shows the %s observation without hiding the goal (%s, %j)', (status, value, comparison, value_format, goal) => {+ render(+ <ArtefactLogList+ reportId="report-1"+ artefacts={[+ makeArtefact(+ {+ title: 'Checkout errors stay low',+ observed_value: value,+ comparison,+ value_format,+ matches: status === 'on track',+ unit: 'errors',+ measured_at: '2026-10-04T12:00:00Z',+ monitoring_ends_at: '2026-10-05T00:00:00Z',+ explanation: 'Interim measurement: legacy wording',+ },+ 'check_observation'+ ),+ ]}+ />+ )+ expect(screen.getByText(status).tagName).toBe('STRONG')+ expect(screen.getByText(`Measured ${value} errors against a goal of ${goal} errors.`)).toBeInTheDocument()+ expect(screen.getByText('12h of monitoring remained at this measurement.')).toBeInTheDocument()+ expect(screen.getAllByText('Interim measurement')).toHaveLength(1)+ expect(screen.queryByText(/legacy wording/)).not.toBeInTheDocument()+ })+it('links the selected predecessors when a replacement starts', () => {const oldPr = 'https://github.com/example/repo/pull/1'render(diff --git a/products/signals/frontend/inbox/components/detail/ArtefactLogList.tsx b/products/signals/frontend/inbox/components/detail/ArtefactLogList.tsxindex 152d379abd3b..cfb1a56b7ae1 100644--- a/products/signals/frontend/inbox/components/detail/ArtefactLogList.tsx+++ b/products/signals/frontend/inbox/components/detail/ArtefactLogList.tsx@@ -53,6 +53,7 @@ import {AutostartSkipContent,CheckLifecycleContent,CheckResultContent,+ CheckObservationContent,CheckScheduledContent,CodeReviewContent,CodeReferenceContent,@@ -80,6 +81,7 @@ import {WorkClaimContent,WorkReleaseContent,} from './artefactTypes'+import { CheckObservationBody } from './CheckObservationBody'import { prActivityTitle } from './prActivityPresentation'import { CHECK_LIFECYCLE_ENTRIES, inconclusiveReasonLabel } from './reportCheckPresentation'@@ -161,6 +163,7 @@ const ARTEFACT_MARKER: Record<string, ComponentType<{ className?: string }>> = {autostart_skip: IconPause,code_review: IconListCheck,check_result: IconCalendar,+ check_observation: IconCalendar,check_scheduled: IconCalendar,check_expired: IconCalendar,check_cancelled: IconCalendar,@@ -651,6 +654,12 @@ function renderArtefactSummary(artefact: SignalReportArtefact): JSX.Element | nu</LemonTag>) : null}+ case 'check_observation':+ return (+ <LemonTag size="small" type="muted">+ Interim measurement+ </LemonTag>+ )case 'check_result': {const outcome = (content as CheckResultContent).outcomeconst meta = outcome ? CHECK_OUTCOME[outcome] : null@@ -785,6 +794,8 @@ function renderArtefactBody({return <AutostartSkipBody content={content as AutostartSkipContent} />case 'code_review':return <CodeReviewBody content={content as CodeReviewContent} />+ case 'check_observation':+ return <CheckObservationBody content={content as CheckObservationContent} />case 'check_result':return <CheckResultBody content={content as CheckResultContent} />case 'check_scheduled':diff --git a/products/signals/frontend/inbox/components/detail/CheckObservationBody.tsx b/products/signals/frontend/inbox/components/detail/CheckObservationBody.tsxnew file mode 100644index 000000000000..977145cca1a5--- /dev/null+++ b/products/signals/frontend/inbox/components/detail/CheckObservationBody.tsx@@ -0,0 +1,63 @@+import { dayjs } from 'lib/dayjs'+import { humanFriendlyDuration } from 'lib/utils/durations'++import { formatReportMetricValue } from '../../utils/reportMetrics'+import type { CheckObservationContent } from './artefactTypes'++export function CheckObservationBody({ content }: { content: CheckObservationContent }): JSX.Element {+ const format = { value_format: content.value_format ?? 'number', unit: content.unit ?? null }+ const goalFormat = {+ ...format,+ value_format: format.value_format === 'count' ? ('number' as const) : format.value_format,+ }+ const comparison = content.comparison+ const bounds = comparison?.operator === 'between' ? comparison.bounds : null+ const thresholds = bounds ? [bounds.lower, bounds.upper] : [comparison?.value]+ const needsPrecision =+ content.matches === false &&+ content.observed_value != null &&+ thresholds.some(+ (threshold) =>+ threshold != null &&+ threshold !== content.observed_value &&+ [format, goalFormat].some(+ (metricFormat) =>+ formatReportMetricValue(metricFormat, threshold) ===+ formatReportMetricValue(metricFormat, content.observed_value)+ )+ )+ const precision = needsPrecision ? 'exact' : undefined+ const goal =+ comparison?.operator === 'between' && comparison.bounds+ ? `between ${formatReportMetricValue(goalFormat, comparison.bounds.lower, precision)} and ${formatReportMetricValue(goalFormat, comparison.bounds.upper, precision)}`+ : comparison?.value != null+ ? `${comparison.operator === 'gte' ? 'at least' : 'at most'} ${formatReportMetricValue(goalFormat, comparison.value, precision)}`+ : null+ const value = formatReportMetricValue(format, content.observed_value, precision)+ const remaining =+ content.monitoring_ends_at && content.measured_at+ ? dayjs(content.monitoring_ends_at).diff(dayjs(content.measured_at), 'second')+ : null++ return (+ <div className="flex w-full flex-col items-start gap-1 text-xs">+ {typeof content.matches === 'boolean' && value !== null && goal !== null ? (+ <>+ <span>+ Goal “{content.title}” is <strong>{content.matches ? 'on track' : 'not on track'}</strong>.+ </span>+ <span>+ Measured {value} against a goal of {goal}.+ </span>+ </>+ ) : (+ <span>{content.explanation?.replace(/^Interim measurement:\s*/, '')}</span>+ )}+ {remaining !== null && Number.isFinite(remaining) && remaining >= 0 ? (+ <span className="text-tertiary">+ {humanFriendlyDuration(remaining, { maxUnits: 2 })} of monitoring remained at this measurement.+ </span>+ ) : null}+ </div>+ )+}diff --git a/products/signals/frontend/inbox/components/detail/artefactTypes.ts b/products/signals/frontend/inbox/components/detail/artefactTypes.tsindex c0ab4f8fefb1..ce344d54c659 100644--- a/products/signals/frontend/inbox/components/detail/artefactTypes.ts+++ b/products/signals/frontend/inbox/components/detail/artefactTypes.ts@@ -1,11 +1,12 @@+import { isObject } from 'lib/utils/guards'// Pure (no-React) artefact helpers shared by the detail logic and the activity-log renderers.// Mirrors the PostHog Desktop inbox's artefact-log domain helpers. Content shapes stay loose// (`Record<string, any>` on the artefact) and are read through these typed accessors so legacy// rows with extra/missing keys never crash a render.--import { isObject } from 'lib/utils/guards'import { identifierToHuman } from 'lib/utils/strings'+import type { ReportMetricApi } from 'products/signals/frontend/generated/api.schemas'+import { SignalReportArtefact } from '../../types'/** Built-in signals pipeline product identifier on a `task_run` artefact. */@@ -123,6 +124,19 @@ export interface CheckResultContent {run_id?: string | null}+export interface CheckObservationContent extends Pick<CheckResultContent, 'title' | 'explanation' | 'observed_value'> {+ measured_at?: string+ monitoring_ends_at?: string | null+ matches?: boolean | null+ value_format?: ReportMetricApi['value_format']+ unit?: string | null+ comparison?: {+ operator: 'lte' | 'gte' | 'between'+ value?: number | null+ bounds?: { lower: number; upper: number } | null+ } | null+}+/** What every entry in a check's life carries. Mirrors `CheckLifecycleEntry` in `artefact_schemas.py`. */export interface CheckLifecycleContent {check_id?: string@@ -346,6 +360,7 @@ export const ARTEFACT_TYPE_LABELS: Record<string, string> = {autostart_skip: 'Work not started',code_review: 'Code review',check_result: 'Follow-up check',+ check_observation: 'Monitoring measurement',check_scheduled: 'Follow-up check scheduled',check_expired: 'Follow-up check expired',check_cancelled: 'Follow-up check cancelled',diff --git a/products/signals/frontend/inbox/logics/inboxReportDetailLogic.test.ts b/products/signals/frontend/inbox/logics/inboxReportDetailLogic.test.tsindex 9ad7fae4c412..0d0f65818abf 100644--- a/products/signals/frontend/inbox/logics/inboxReportDetailLogic.test.ts+++ b/products/signals/frontend/inbox/logics/inboxReportDetailLogic.test.ts@@ -1,3 +1,5 @@+import { MOCK_DEFAULT_PROJECT, MOCK_DEFAULT_TEAM } from 'lib/api.mock'+import { waitFor } from '@testing-library/react'import { expectLogic } from 'kea-test-utils'import posthog from 'posthog-js'@@ -14,7 +16,7 @@ import type { SignalReportCheckApi } from 'products/signals/frontend/generated/aimport { ReportTaskPurpose } from '../components/detail/artefactTypes'import { INBOX_EVENTS } from '../inboxAnalytics'import { inboxSceneLogic } from '../inboxSceneLogic'-import { EnrichedReviewer, SignalReport } from '../types'+import { EnrichedReviewer, SignalReport, SignalReportStatus } from '../types'import { inboxBulkActionsLogic } from './inboxBulkActionsLogic'import { ReportTaskEntry, implementationSlotClaim, inboxReportDetailLogic } from './inboxReportDetailLogic'@@ -224,6 +226,89 @@ describe('inboxReportDetailLogic', () => {logic.unmount()})+ it('finishes slow pagination across polls and retains project-scoped activity, task runs, and reviewers', async () => {+ const observation = {+ id: 'observation-1',+ type: 'check_observation',+ content: {},+ created_at: '2026-10-01T00:00:00Z',+ }+ const taskRun = {+ id: 'task-run-1',+ type: 'task_run',+ content: { task_id: 'task-1', product: 'signals', type: 'implementation' },+ created_at: '2026-01-01T00:00:00Z',+ }+ const note = {+ id: 'note-1',+ type: 'note',+ content: { note: 'Earlier activity' },+ created_at: '2026-01-01T00:00:00Z',+ }+ let finishPage: () => void = () => {}+ const pendingPage = new Promise<void>((resolve) => {+ finishPage = resolve+ })+ const requests: string[] = []+ const fetchPage = jest.fn(async ({ request }: { request: Request }) => {+ const url = new URL(request.url)+ requests.push(url.pathname)+ if (url.searchParams.has('offset')) {+ await pendingPage+ return { results: [artefact, taskRun, note], next: null }+ }+ return {+ results: [observation],+ next: `/api/projects/${MOCK_DEFAULT_PROJECT.id}/signals/reports/report-1/artefacts/?limit=1000&offset=1000`,+ }+ })+ useMocks({+ get: {+ '/api/projects/:team_id/signals/reports/:id/artefacts/': fetchPage,+ '/api/projects/:team_id/tasks/:id/': {+ id: 'task-1',+ latest_run: { status: TaskRunStatus.COMPLETED },+ },+ },+ })+ logic.unmount()+ jest.useFakeTimers()+ initKeaTests(true, { ...MOCK_DEFAULT_TEAM, id: 99, project_id: MOCK_DEFAULT_PROJECT.id })+ logic = inboxReportDetailLogic({+ reportId: REPORT.id,+ report: { ...REPORT, status: SignalReportStatus.IN_PROGRESS },+ })+ logic.mount()+ try {+ await waitFor(() => expect(fetchPage).toHaveBeenCalledTimes(2))+ await jest.advanceTimersByTimeAsync(10000)+ expect(fetchPage).toHaveBeenCalledTimes(2)+ expect(logic.values.reportArtefactsLoading).toBe(true)+ expect(requests).toEqual([+ `/api/projects/${MOCK_DEFAULT_PROJECT.id}/signals/reports/${REPORT.id}/artefacts/`,+ `/api/projects/${MOCK_DEFAULT_PROJECT.id}/signals/reports/${REPORT.id}/artefacts/`,+ ])+ finishPage()+ await expectLogic(logic).toFinishAllListeners()+ expect(logic.values.reportArtefacts?.map((row) => row.id)).toEqual([+ 'observation-1',+ 'reviewers-1',+ 'task-run-1',+ 'note-1',+ ])+ expect(logic.values.reportReviewers).toEqual([reviewer])+ expect(logic.values.reportTasks?.map((entry) => entry.task.id)).toEqual(['task-1'])+ expect(logic.values.reportArtefactsLoading).toBe(false)+ await jest.advanceTimersByTimeAsync(5000)+ await expectLogic(logic).toFinishAllListeners()+ expect(fetchPage).toHaveBeenCalledTimes(4)+ } finally {+ finishPage()+ logic.unmount()+ jest.useRealTimers()+ }+ })+it('keeps a removed reviewer hidden until the refreshed list arrives', async () => {let releaseRefresh: () => void = () => {}const heldRefresh = new Promise<void>((resolve) => {diff --git a/products/signals/frontend/inbox/logics/inboxReportDetailLogic.ts b/products/signals/frontend/inbox/logics/inboxReportDetailLogic.tsindex 35a91b5456e8..3e16cc425e55 100644--- a/products/signals/frontend/inbox/logics/inboxReportDetailLogic.ts+++ b/products/signals/frontend/inbox/logics/inboxReportDetailLogic.ts@@ -16,7 +16,7 @@ import { loaders } from 'kea-loaders'import { lemonToast } from '@posthog/lemon-ui'-import api from 'lib/api'+import api, { ApiConfig } from 'lib/api'import { ApiError } from 'lib/api-error'import { dayjs } from 'lib/dayjs'import { SignalNode } from 'scenes/debug/signals/types'@@ -28,6 +28,7 @@ import { userLogic } from 'scenes/userLogic'import { Task, TaskRunStatus } from 'products/posthog_ai/frontend/types/taskTypes'import {signalsReportArtefactsDiff,+ signalsReportArtefactsList,signalsReportChecksApproveCreate,signalsReportChecksDestroy,signalsReportChecksList,@@ -45,6 +46,7 @@ import {} from 'products/signals/frontend/generated/api'import type {CommitDiffResponseApi,+ PaginatedSignalReportArtefactListApi,PullRequestCheckApi,PullRequestCommentApi,PullRequestCommentReactionApi,@@ -77,13 +79,7 @@ import {} from '../inboxAnalytics'import { inboxSceneLogic } from '../inboxSceneLogic'import { inboxTaskKickoffLogic } from '../inboxTaskKickoffLogic'-import {- EnrichedReviewer,- SignalReport,- SignalReportArtefact,- SignalReportArtefactResponse,- SignalReportStatus,-} from '../types'+import { EnrichedReviewer, SignalReport, SignalReportArtefact, SignalReportStatus } from '../types'import { ChartPlacements, resolveChartPlacements } from '../utils/chartPlacement'import { reportPullRequests, primaryReportPullRequest, type ReportPullRequest } from '../utils/reportPullRequests'import { inboxBulkActionsLogic } from './inboxBulkActionsLogic'@@ -487,7 +483,7 @@ export interface inboxReportDetailLogicActions {prComments: readonly PullRequestCommentApi[] | nullpayload?: void}- loadReportArtefacts: () => any+ loadReportArtefacts: (_: void) => voidloadReportArtefactsFailure: (error: string,errorObject?: any@@ -497,10 +493,10 @@ export interface inboxReportDetailLogicActions {}loadReportArtefactsSuccess: (reportArtefacts: SignalReportArtefact[],- payload?: any+ payload?: void) => {reportArtefacts: SignalReportArtefact[]- payload?: any+ payload?: void}loadReportChecks: (_: void) => voidloadReportChecksFailure: (@@ -832,11 +828,23 @@ export const inboxReportDetailLogic = kea<inboxReportDetailLogicType>([reportArtefacts: [null as SignalReportArtefact[] | null,{- loadReportArtefacts: async () => {- const response: SignalReportArtefactResponse = await api.signalReports.artefacts(props.reportId, {- limit: ARTEFACT_FETCH_LIMIT,- })- return response.results+ loadReportArtefacts: async (_: void, breakpoint): Promise<SignalReportArtefact[]> => {+ let response = await signalsReportArtefactsList(+ String(ApiConfig.getCurrentProjectId()),+ props.reportId,+ {+ limit: ARTEFACT_FETCH_LIMIT,+ }+ )+ await breakpoint()+ const artefacts = [...response.results]+ while (response.next) {+ // nosemgrep: prefer-codegen-api -- follows the server-provided pagination next link.+ response = await api.get<PaginatedSignalReportArtefactListApi>(response.next)+ await breakpoint()+ artefacts.push(...response.results)+ }+ return artefacts as SignalReportArtefact[]},},],@@ -1958,7 +1966,7 @@ export const inboxReportDetailLogic = kea<inboxReportDetailLogicType>([// Auto-disposed on unmount / hidden tab.cache.disposables.add(() => {const interval = setInterval(() => {- if (!values.shouldPollReportTasks) {+ if (!values.shouldPollReportTasks || values.reportArtefactsLoading) {return}actions.loadReportArtefacts()diff --git a/products/signals/frontend/inbox/utils/reportMetrics.ts b/products/signals/frontend/inbox/utils/reportMetrics.tsindex fcf9e77e4436..d6310f4297a8 100644--- a/products/signals/frontend/inbox/utils/reportMetrics.ts+++ b/products/signals/frontend/inbox/utils/reportMetrics.ts@@ -85,7 +85,8 @@ export interface ReportMetricValueParts {export function formatReportMetricParts(metric: ReportMetricFormatting,- value: number | null | undefined+ value: number | null | undefined,+ precision?: 'exact'): ReportMetricValueParts | null {if (value === null || value === undefined || !Number.isFinite(value)) {return null@@ -94,6 +95,16 @@ export function formatReportMetricParts(const valueFormat = metric.value_format ?? 'number'const unit = metric.unit ?? null+ if (precision === 'exact') {+ if (valueFormat === 'percentage' || valueFormat === 'percentage_scaled') {+ return {+ value: `${valueFormat === 'percentage_scaled' ? value * 100 : value}%`,+ unit: unit === '%' ? null : unit,+ }+ }+ return { value: String(value), unit }+ }+switch (valueFormat) {case 'count':return { value: humanFriendlyNumber(value, 0), unit }@@ -126,9 +137,10 @@ export function formatReportMetricParts(export function formatReportMetricValue(metric: ReportMetricFormatting,- value: number | null | undefined+ value: number | null | undefined,+ precision?: 'exact'): string | null {- const parts = formatReportMetricParts(metric, value)+ const parts = formatReportMetricParts(metric, value, precision)return parts ? withUnit(parts.value, parts.unit) : null}diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.tsindex e3c7af931991..10c963d327f0 100644--- a/services/mcp/src/api/generated.ts+++ b/services/mcp/src/api/generated.ts@@ -74885,6 +74885,7 @@ export namespace Schemas {* * `work_release` - Work Release* * `pull_request` - Pull Request* * `check_result` - Check Result+ * * `check_observation` - Check Observation* * `check_scheduled` - Check Scheduled* * `check_expired` - Check Expired* * `check_cancelled` - Check Cancelled@@ -74923,6 +74924,7 @@ export namespace Schemas {WorkRelease: 'work_release',PullRequest: 'pull_request',CheckResult: 'check_result',+ CheckObservation: 'check_observation',CheckScheduled: 'check_scheduled',CheckExpired: 'check_expired',CheckCancelled: 'check_cancelled',diff --git a/services/mcp/src/generated/signals/api.ts b/services/mcp/src/generated/signals/api.tsindex 3d4f495183f6..d66d590effcd 100644--- a/services/mcp/src/generated/signals/api.ts+++ b/services/mcp/src/generated/signals/api.ts@@ -727,7 +727,7 @@ export const SignalsReportArtefactsPartialUpdateBody = () => zod)/**- * Delete an artefact, addressed by id. Deleting the latest row of a status type reverts the report's canonical status to the previous version (latest-wins over what remains). `task_run` artefacts are an append-only work log and cannot be deleted. Neither can the types this API cannot write, which the pipeline owns: `autostart_skip`, `check_cancelled`, `check_expired`, `check_result`, `check_scheduled`, `code_review`, `impact_measurement_plan`, `implementation_decision`, `implementation_dispatch`, `implementation_handover`, `implementation_replacement`, `pull_request`, `ranking_score`, `report_link`, `summary_change`, `task_run`, `title_change`, `video_segment`, `work_claim`, `work_release`.+ * Delete an artefact, addressed by id. Deleting the latest row of a status type reverts the report's canonical status to the previous version (latest-wins over what remains). `task_run` artefacts are an append-only work log and cannot be deleted. Neither can the types this API cannot write, which the pipeline owns: `autostart_skip`, `check_cancelled`, `check_expired`, `check_observation`, `check_result`, `check_scheduled`, `code_review`, `impact_measurement_plan`, `implementation_decision`, `implementation_dispatch`, `implementation_handover`, `implementation_replacement`, `pull_request`, `ranking_score`, `report_link`, `summary_change`, `task_run`, `title_change`, `video_segment`, `work_claim`, `work_release`.* @summary Delete an artefact*/export const SignalsReportArtefactsDestroyParams = () => zod.object({